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A Systematic Evidence Review 


Summary 


Nonacademic Interventions for Postsecondary Enrollment and Success 
with Rural and High-Poverty Populations: A Systematic Evidence Review 


Student success in postsecondary education depends on academic preparation and on having the 
nonacademic knowledge, skills, and behaviors to successfully navigate the demands of postsecondary 
study. While nonacademic competencies are important for all students, rural and high-poverty 
populations face unique nonacademic challenges, such as access to adequate college counseling and 
mentoring. They also have lower postsecondary enrollment and completion rates than their nonrural 

or low-poverty peers (Adelman, 2006; Byun et al., 2012; Hu, 2003). This review sought to identify 
nonacademic interventions with evidence of positive effects on postsecondary enrollment, academic 
performance, persistence, and completion, particularly with rural and high-poverty populations. Results 
identified five interventions with potentially positive effects for all students: Free Application for Federal 
Student Aid (FAFSA) interventions, the Facilitating Long-term Improvements in Graduation and Higher 
Education for Tomorrow (FLIGHT) program, college counseling, summer counseling, and summer bridge 
programs. Of these five interventions, two (FLIGHT and summer counseling) showed positive effects with 
high-poverty populations. In addition to these five interventions, Upward Bound showed positive effects 
for rural and high-poverty students, but not for the overall study population. Additionally, the Upward 
Bound study took place between 1992 and 1994. The program has changed substantially since that time, 
and thus the findings of the study may not be generalizable to current conditions. These results indicate 
a need for additional high-quality research with rural and high-poverty populations so that education 
leaders can select and implement interventions that work in their contexts. 


Key findings 
* There is a medium to large extent of evidence for the effect of FAFSA interventions on 
postsecondary enrollment. Additionally, there is a small extent of evidence for the effects of FAFSA, 


FLIGHT, college counseling, summer counseling, and summer bridge programs on at least one metric 
of postsecondary enrollment or success. 


* Only one study found positive effects for rural populations, and three found positive effects 
for high-poverty populations. Upward Bound reported positive effects for students in rural and 
high-poverty schools, and Upward Bound, summer counseling, and FLIGHT reported positive effects 
on high-poverty populations. Based on our review of research from key databases, there appears 
to be a gap in rigorous evidence about nonacademic interventions with rural and high-poverty 
populations. 


* The information available on the effects of nonacademic interventions on postsecondary 
enrollment and success appears to have significant gaps. Our search of the published literature 
returned only 17 studies (across nine interventions) that used well-designed and well-implemented 
experimental or quasi-experimental designs, and only eight of these studies (across six interventions) 
found statistically significant positive effects of nonacademic interventions on postsecondary 
outcomes. 


Nonacademic Interventions for Postsecondary Enrollment and Success with Rural and High-Poverty Populations: 1 
A Systematic Evidence Review 


Background on this Research Review 


Success in postsecondary education depends not only on academic preparation but also on successfully 
navigating the social, cultural, and other implicit demands of postsecondary study. For example, enrolling 
in college may require students to get immunizations and to complete college entrance exams, financial 
aid paperwork, and housing forms. Students must be aware of these requirements and have the tools and 
knowledge necessary to meet them. Likewise, continued enrollment and success in college can depend 
on students’ persistence or grit, their connections to the college community, time management skills, and 
ability to acclimate to new cultural norms. Finally, successful completion of a degree requires students to 
take appropriate courses, balance the demands of paid work with course requirements, seek help when 
necessary, and continue to prioritize a college degree over more immediate career options. Interventions 
at the high school or even middle and elementary school levels may prepare students to overcome these 
hurdles by building their nonacademic competencies, thus improving their likelihood of postsecondary 
enrollment and success. 


While nonacademic competencies are important for all students, rural populations and high-poverty 
populations in particular may benefit from interventions that address them because these populations enroll 
in and complete college at significantly lower rates than their nonrural or low-poverty peers (Adelman, 2006; 
Byun et al., 2012; Hu, 2003). Although academic competencies contribute to this gap, rural and high-poverty 
populations also face unique nonacademic challenges to postsecondary enrollment and success. For example, 
rural students are less likely to encounter college-educated role models (Castleman & Page, 2014; Hoxby & 
Avery, 2013) who could help them understand the steps needed to enroll or prepare them for the challenges 
in persisting in postsecondary education. Likewise, high-poverty students often face inadequate college 


Nonacademic Interventions for Postsecondary Enrollment and Success with Rural and High-Poverty Populations: 2 
A Systematic Evidence Review 


counseling at their schools (Plank & Jordan, 2001). Nonacademic interventions could connect rural and high- 
poverty students with mentors or outside counselors to fill these gaps. Additionally, the American Council on 
Education has shown that the majority of public college students attend a postsecondary institution within 

50 miles of home (Hillman & Weichman, 2016), and rural and high-poverty students are less likely to apply to 
and enroll in college if few colleges are located nearby (Rouse, 1995; Turley, 2009). Nonacademic interventions 
could address the challenges of identifying colleges outside students’ immediate surroundings and help them 
overcome barriers to moving and living far from home. 


In keeping with this research, education leaders from four partner school districts in eastern Kentucky 
report that many of their students deemed academically well prepared nonetheless fail to enroll 

in college or leave college after a year and return home. These stakeholders expressed a need to 
implement nonacademic interventions to improve postsecondary enrollment and success for their 
rural high-poverty populations. 


When selecting an intervention to implement, stakeholders in Kentucky, as well as those nationwide 

who work with rural and high-poverty populations, need evidence that the intervention will work in their 
particular context. As mentioned above, rural and high-poverty populations face unique challenges, and 
an intervention with evidence in urban or low-poverty areas may not be easily transferrable to a rural 

or high-poverty area. Recognizing this reality, policymakers and program leaders seek interventions that 
demonstrate effectiveness in contexts similar to their own, consistent with Every Student Succeeds Act 
(ESSA) requirements (Institute of Education Sciences, 2019). Thus, a systematic review of the evidence for 
the effectiveness of nonacademic interventions on rural and high-poverty populations would be valuable 
to stakeholders working with these populations. 


At this time, there is no repository of evidence on which rural, high-poverty stakeholders can draw 

that identifies the effects of nonacademic interventions on postsecondary outcomes in their locales. 
While a practice guide on the path to college published by the Institute of Education Sciences 

highlights key postsecondary enrollment interventions and makes recommendations focused on 
nonacademic competencies, it does not discuss the promise of such interventions for rural high-poverty 
populations in particular, nor has it been updated since 2009 (Tierney et al., 2009). Likewise, the What 
Works Clearinghouse (WWC) website can be searched for a variety of secondary and postsecondary 
interventions, but it is not searchable or sortable by population, nor can it be narrowed to nonacademic 
interventions in particular. 


This dearth of easily accessible evidence presents a challenge for policymakers and school and district 
personnel who need to identify interventions that can be implemented in their particular contexts. This 
evidence review was designed to fill that need. See box 1 for a list of key terms used in this review and 
box 2 for a description of the methods and sample. 
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[Et Box 1: Key terms 


Intervention. A program, strategy, or activity designed to improve student outcomes. An intervention 
may address academic competencies, nonacademic competencies, or both. School, district, or external 
staff (for example, from nonprofit organizations, publishers, or technology companies) actively design 
and/or implement interventions. This definition excludes whole-school reforms, which usually encompass 
multiple interventions. 


Nonacademic competencies. The knowledge, skills, and behaviors that enable students to navigate 
the social, cultural, and other implicit demands of postsecondary study. These include competencies 
that are not explicitly tested or required by postsecondary institutions or courses but are necessary 
to enrolling, succeeding, persisting in, and completing postsecondary education (for example, time 
management, a sense of belonging or connectedness, or knowledge of degree requirements). For 
additional information on the research base for nonacademic competencies, see appendix A. 


Nonacademic interventions. Interventions including programmatic supports or activities that 

address at least one nonacademic competency. A nonacademic intervention may address, for 

example, knowledge of the postsecondary enrollment and registration process, the challenges of living 
independently for the first time, adapting to cultural norms of dorm living, or strengthening students’ 
connections to the college community. For inclusion in this review, an intervention must address at least 
one nonacademic competency. 


What Works Clearinghouse. The What Works Clearinghouse (WWC) examines the available research on 
educational programs, policies, and practices. Trained reviewers examine the research design, methods, 
and reported outcomes in research studies and provide ratings of the strength of the design and 
implementation of each study as well as the size of the reported impact. The WWC also systematically 
searches for studies on programs, policies, and practices and consolidates the evidence to provide 
ratings of the average impact and the extent of evidence for each intervention. 


Methodological terms 


Does not meet WWC standards. This study rating indicates that a study does not establish a 
relationship between the intervention and outcomes. 


Improvement index. This index describes the effect of an intervention on outcomes in percentile units. 
It can be interpreted as the change in percentile rank attributed to the intervention for an average 
student. An improvement index of +1 indicates that the intervention would move the average student 
from the 50th to the 51st percentile. 


Medium to large extent of evidence. An intervention with a medium to large extent of evidence 
has evidence from at least two studies, in at least two schools, and with more than 350 students or 
14 classrooms. 
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Meets WWC standards with reservations. This rating indicates that a study has established a 
relationship between the intervention and outcomes, but that causality is not firmly established. 
Additional research is needed to confirm that the intervention causes the outcomes. A study may 
receive this rating if it is a randomized controlled trial that was not well implemented or if it used a 
strong quasi-experimental design. 


Meets WWC standards without reservations. This WWC study rating indicates that a study used 
rigorous methods that allow the reader to conclude that the intervention caused the outcomes. In this 
report, studies of nonacademic interventions meeting this criterion indicate that, under the conditions 
used in the study, the nonacademic intervention caused changes in postsecondary enrollment, 
academic performance, persistence, or completion. 


Mixed effects. Interventions with mixed effects are those where some studies show positive effects 
and others show negative effects, or where the number of studies with positive or negative effects is 
outweighed by the number of studies with no effects. 


Positive effects. A study with an estimated effect that is positive and statistically significant 
demonstrates “positive effects.” The term applies to findings from a study. 


Potentially positive effects. A term the WWC uses to characterize study findings. Interventions with 
potentially positive effects have at least one study of the intervention that shows statistically significant 
positive effects, no studies that show statistically significant negative effects, and the number of studies 
with no effects is smaller than the number of studies with positive effects. 


Small extent of evidence. An intervention with a small extent of evidence has evidence from only one 
study, one school, or fewer than 350 students and 14 classrooms. 


Well-designed and well-implemented studies. An experimental study is “well-designed and well- 
implemented” if it meets WWC standards without reservations, and a quasi-experimental study is “well- 
designed and well-implemented” if it meets WWC standards with reservations. 


Outcomes 


Postsecondary academic performance. Student test scores, grades, passing rates, or other course 
performance indicators in postsecondary education. 


Postsecondary completion. Student completion of a degree program or graduation from a degree- 
granting institution. This includes associate’s and bachelor’s degrees but does not include workforce 
training or certification programs that do not award a degree. 


Postsecondary education. For the purposes of this report, postsecondary education includes 
both two-year and four-year degree-granting institutions. It does not include training programs or 
industry certifications. 
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Postsecondary enrollment. Student initial enrollment or matriculation at a postsecondary degree- 
granting institution within one year of graduating from high school. 


Postsecondary persistence or retention. Student continued enrollment and credit accrual at a degree- 
granting institution beyond the first semester of the first year. 


Postsecondary enrollment and success. In this report, postsecondary enrollment and success is an 
umbrella term encompassing postsecondary enrollment, academic performance, persistence, and 
completion. 


Population characteristics 


High-poverty populations. This review defines high poverty as interventions wherein more than 50 
percent of the students are eligible for free or reduced-price lunch. High-poverty populations are the 
students who participate in such interventions. 


Rural populations. This review uses the NCES definition of rural locales, including all three rural locale 
designations: Rural - Fringe (locale 41), Rural - Distant (locale 42), and Rural - Remote (locale 43). Rural 
populations are students living and attending school in rural locales. 


Sources: Karp, 2011; NCES, 2019a; 201 9b; https://ies.ed.gov/ncee/wwe and https://ies.ed.gov/ncee/wwc/Glossary 


This review addressed three primary questions: 


1.What is the existing evidence for positive effects of nonacademic interventions on student 
postsecondary enrollment, academic performance, persistence, and completion for all students from 
well-designed and well-implemented experimental or quasi-experimental studies? 


2.What is the existing evidence for positive effects of nonacademic interventions on student 
postsecondary outcomes for rural and high-poverty populations? 


3.What additional research is needed to address the evidence gap for postsecondary student 
outcomes with rural and high-poverty populations? 
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Box 2: Data sources, sample, and methods 
This review used a systematic evidence review to address the three primary questions. 


Data sources. The review drew on published research in scholarly journals, dissertations, What Works 
Clearinghouse (WWC) intervention reports, and WWC reviews of individual studies. The first step was 

a search of ProQuest’s Education Database to systematically identify published research in scholarly 
journals, book chapters, commissioned reports, conference papers, print media, and dissertations. 

The second step was a search focused on WWC intervention reports and WWC reviews of individual 
studies to identify additional studies that met inclusion criteria and were not previously identified in the 
ProQuest search. 


Sample. The sample for this review included all studies published between 1998 and 2018 that met 
the inclusion criteria discussed below. Broadly speaking, a study must have evaluated a nonacademic 
intervention for students, ages 5 to 19, designed to improve postsecondary enrollment, academic 
performance, persistence, or completion. The interventions discussed in this review also had to have 
at least one study that met WWC design standards and showed positive effects of the intervention on 
postsecondary enrollment, academic performance, persistence, or completion. 


Inclusion criteria. To be included in this evidence review, a study must meet two sets of criteria. First, 
the study must examine nonacademic interventions implemented with the relevant population and 
examine the outcomes of interest. A study must: 


1.Include original research. Reviewers excluded studies summarizing other research, news articles, 
and promotional material for interventions. When the search returned literature reviews, meta- 
analyses, and WWC intervention reports, the research team reviewed the sources in these 
documents and screened them for the inclusion criteria. 


2. Test the effects of an intervention or policy. The review excluded studies focused solely on the relationship 
between nonacademic competencies themselves (such as grit or persistence) and student outcomes or 
the relationship between demographic background variables (such as socioeconomic status, race, and 
parental education) and student outcomes. The purpose of this criterion was to ensure that the evidence 
review was focused on programs and policies that could be implemented by stakeholders in their schools, 
districts, or states. 


3.Include participants who were ages 5 to 19 years at the time of the intervention, excluding any 
interventions that took place entirely with postsecondary students at postsecondary institutions. 
This criterion ensured that the evidence review was focused on interventions that could be 
implemented with students in K-12 public schools or during the summer after high school. 


4.Examine an intervention with at least one nonacademic component (see box 1). The review excluded 
interventions that were purely academically focused. 


5.Measure one of the four outcomes of interest: postsecondary enrollment; postsecondary academic 
performance; postsecondary persistence; or postsecondary completion. 
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6. Have been published in English between January 1, 1998, and December 20, 2018 (the date the 
searches were completed). This criterion maximized the likelihood that authors would report all 
the information needed for a WWC review. ' 


Second, to be included in this review, a study must be well-designed and well-implemented and show 
positive effects. The study must: 


7.Meet WWC standards (with or without reservations). This eliminated studies using qualitative designs 
and those using less rigorous quantitative methods, such as correlational and descriptive studies. 


8.Report statistically significant or substantively important positive effects (see definitions of these 
terms in box 1). 


Methods. The research team searched ProQuest’s Education Database and the WWC website for studies to 
include in the sample. The team searched ProQuest's Education Database using a series of predetermined 
search terms and created a list of all studies ProQuest returned for each search term. The research team 

also examined both the “Path to Graduation” and “Postsecondary” topic areas in the intervention reports and 
reviews of individual studies sections of the WWC website. If the research team found a study in ProQuest's 
Education Database that the WWC had already reviewed, the team did not re-review the study but instead used 
the ratings recorded on the WWC website. The team created a list of all studies included in the intervention 
reports and individual study reviews in these topic areas that included postsecondary outcomes. The studies 
found either in ProQuest or on the WWC website formed the full sample of studies screened for this review. 


For each study in this sample, researchers completed an online screening form, documenting whether 

the study met the first six inclusion criteria, regarding the focus on nonacademic interventions with the 
relevant populations and outcomes. Next, for each study that met these inclusion criteria but had not been 
reviewed by the WWC, the study team carried out a full WWC review, including querying authors when 
needed. The researchers involved in this step were all WWC certified to review studies using version 3.0 of 
the WWC standards (the version current at the start of the review process). Studies previously reviewed by 
the WWC were not reviewed again; rather, this review simply documented the ratings the WWC assigned. 
Finally, the research team narrowed the list of studies to those that met WWC standards with or without 
reservations and demonstrated positive effects on one of the outcomes of interest (the final two inclusion 
criteria, above). This review documents the findings of the studies that met all eight inclusion criteria. 


For additional detail on the searching and screening process, including training and reliability monitoring in 
the screening process, WWC author queries, and the list of search terms and search results, see appendix B. 
Appendix C reproduces the form researchers used to screen studies for inclusion in this review. Appendix 
D provides the full list of studies that examined nonacademic interventions with the relevant population 
and outcomes and were screened for design and effects. Appendix E provides a full description of the 
interventions with studies that met all the initial inclusion criteria and also used designs that met WWC 
standards with or without reservations. 


' The WWC was established in 2002, with its standards reflecting a move toward more rigorous study designs and reporting that was already under 
way in education research. This review extended its search window back to 1998 to capture some of these pre-WWC studies. 
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Findings from the Review 


The search process yielded 1,777 studies. Of these, 1,712 did not focus on the postsecondary outcomes 

of nonacademic interventions with students ages 5 to 19.2 Of the 65 studies that focused on the relevant 
population and outcomes, 48 did not meet WWC standards (figure 1). A total of eight studies investigating six 
interventions met all inclusion criteria, including the requirement that the studies meet WWC standards with 
or without reservations and show statistically significant positive effects. Only one of these six interventions 
showed positive effects for rural populations (Seftor et al., 2009), and only three showed positive effects 
specifically for high-poverty populations (Bettinger et al., 2012; Carrell & Sacerdote, 2013; Castleman et al., 
2014; Murphy et al., 2010; Page et al., 2018; Philp, 2015; Seftor et al., 2009). 


The next section of this review describes each intervention that had effects on postsecondary outcomes 
for the overall study population. Next, it discusses interventions that had effects specifically for rural and 
high-poverty populations. For a more detailed description of the interventions, see appendix E. 


Figure 1. A search of ProQuest's Education Database and the What Works Clearinghouse (WWC) website 
returned 1,777 unique studies; only 65 focused on nonacademiic interventions with the relevant populations and 
outcomes, and only eight studies met WWC standards and had positive effects on postsecondary outcomes 


Searches returned 1,777 unique studies 


65 focused on the postsecondary (PS) 
outcomes of nonacademic interventions 
with students ages 5 to 19 


17 met WWC standards 


8 had positive 


PS is postsecondary. 
Source: Authors’ review of literature identified using ProQuest’s Education Database and the WWC website between June and December 2018. 


? Interventions taking place in preschool or early elementary school in which students are tracked into postsecondary education are rare. 
However, we included students as young as 5 to allow for the inclusion of this type of intervention. Interventions that took place entirely with 
postsecondary students in a postsecondary institution were excluded. See appendix B for additional information. 
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This review identified five nonacademic interventions with potentially 
positive effects on at least one postsecondary outcome for the overall 
study population.’ 


The eight studies that met all inclusion criteria investigated six interventions.* Of these six interventions, 
five have evidence of positive effects on postsecondary outcomes for the overall study population, based 
on well-designed and well-implemented studies: support for completing the Free Application for Federal 
Student Aid (FAFSA), Facilitating Long-Term Improvements in Graduation and Higher Education for 
Tomorrow (FLIGHT), college counseling, summer counseling, and summer bridge programs (table 1).° This 
section of the review briefly describes each intervention and summarizes the findings on their effects on 
postsecondary outcomes. For a more detailed description of the interventions, see appendix E. 


FAFSA interventions showed positive effects on postsecondary enrollment and persistence 


FAFSA interventions provide students and families with information on the importance of completing the 
FAFSA to obtain college aid, send reminders on key financial aid deadlines, and may assist low-income 
families in completing the application. These interventions can seek to engage high-poverty populations 
in particular, but may also be implemented broadly, for all students who have expressed an interest in 
postsecondary education. FAFSA interventions have the potential to overcome the particular barriers 
faced by rural and high-poverty populations by giving them information about the college application 
process to which they might otherwise have no access. 


This review identified two studies of FAFSA interventions that met WWC standards (Bettinger et al., 2012, 
and Page et al., 2018). Both showed positive, statistically significant effects of FAFSA interventions on 
students’ enrollment in postsecondary education. The two studies have an average improvement index 
of +8, indicating that, on average, FAFSA interventions can move the average student from the 50th to 
the 58th percentile in the distribution of students enrolling in postsecondary education. With two studies 
including more than 350 students and more than two schools, the extent of evidence of the effect of 
FAFSA information interventions on postsecondary enrollment is medium to large (WWC, 2013a, 2019a). 


Additionally, one study of a FAFSA intervention that included both information about financial aid and 
support for completing the FAFSA showed a positive effect of the intervention on student persistence in 
college for two consecutive years (Bettinger et al., 2012). The improvement index for this outcome was +9, 
indicating that FAFSA interventions can move the average student from the 50th to the 59th percentile in 
the distribution of students persisting in college. The extent of evidence for postsecondary persistence is 
small because only one study that met WWC standards examined this outcome (table 1). 


? Upward Bound demonstrated potentially positive effects for rural and high-poverty students (Seftor et al., 2009), but not for the overall study 
population (Myers et al., 2004). Thus, this intervention is discussed in the next section. 


* The WWC had already reviewed all eight studies discussed in this review. The search process did return additional studies that had not been 
reviewed by the WWC, but none met the criteria for inclusion in this review. See appendixes B and D for more information. 


° This review identified no nonacademic interventions implemented with students ages 5 to 19 that had high-quality evidence of effects on 
students’ postsecondary academic performance. 
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FLIGHT showed positive effects on postsecondary enrollment 


FLIGHT is a school-based program that provides mentoring and other supports to middle and high school 
students to improve their chances of enrolling and succeeding in college. The program seeks to engage 
students who qualify for free or reduced-price lunch and are in good academic and disciplinary standing 
(WWC, 2019b). FLIGHT includes three key nonacademic services: wraparound case management from 
school staff, including meetings to review and monitor students’ academic progress and steps toward 
college; one-on-one student mentoring from volunteers; and a detailed annual report, provided to 
parents, teachers, and guidance counselors, of students’ grades, attendance, and test scores (Philp, 2015). 
Results showed that FLIGHT has a positive, statistically significant effect on postsecondary enrollment. 
While the extent of the evidence is small, stemming from only one study, the average improvement index 
is +39, suggesting that FLIGHT can move the average student from the 50th to the 89th percentile in the 
distribution of students enrolling in postsecondary education. 


College counseling interventions showed potentially positive effects on postsecondary enrollment® 


College counseling interventions provide high school seniors with mentoring and assistance with completing 
college applications. In the college counseling intervention that met inclusion standards for this report, 
college undergraduates met with high school seniors to help them navigate the steps involved in applying 

to college, provided feedback on their application essays, and helped them fill out paperwork, including 
financial aid applications. The intervention also included financial support with application fees and a $100 
incentive for completing the application process. The population of interest was students with high grade 10 
test scores who had not yet applied to college in their senior year of high school (WWC, 2013b). 


One study of a college counseling intervention in New Hampshire (Carrell & Sacerdote, 2013)’ found 
positive effects on postsecondary enrollment, with an improvement index of +5, indicating that college 
counseling can move the average student from the 50th to the 55th percentile in the distribution of 
students enrolling in postsecondary education. The study also examined postsecondary persistence but 
found no statistically significant effects on this outcome. 


Summer counseling interventions showed mixed effects on postsecondary enrollment and potentially 
positive effects on postsecondary persistence 


Summer counseling broadly refers to programs aimed at ensuring that high school graduates 
successfully matriculate in college in the fall after high school graduation. Summer counseling supports 
students during the summer after high school in getting through the steps necessary to matriculate 
and in overcoming barriers to postsecondary enrollment. The specific services and delivery of summer 
counseling programs vary but may include text messaging, in-person meetings, and social media 


6 In the WWC systematic review of college counseling, only one study met WWC standards with or without reservations (WWC, 20136). This study 
demonstrated a positive effect on postsecondary enrollment (Carrell & Sacerdote, 2013). The WWC characterized college counseling as having 
potentially positive effects on postsecondary enrollment. 

’ The WWC reviewed a version of this working paper entitled “Late interventions matter too: The case of college coaching New Hampshire.” Since this 
review, Carrell and Sacerdote have updated the paper with a new title (‘Why do college going interventions work”) and replaced the original copy. 
Regional Educational Laboratory Appalachia analysts examined both the WWC intervention report and the full study as currently published. 
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campaigns to offer “information about tasks required for postsecondary enrollment, as well as assistance 
in overcoming unanticipated financial, informational, and socioemotional barriers” (WWC, 2018, p. 1). 


Results showed mixed effects of summer counseling on postsecondary enrollment (WWC, 2018). One 

study of summer counseling showed statistically significant positive effects on postsecondary enrollment, 
resulting in its inclusion in this review (Castleman et al., 2014), and four other studies showed no effects. 
The average improvement index across all five studies was +5, which corresponds to moving the average 
student from the 50th to the 55th percentile in the distribution of students enrolling in postsecondary 
education. However, this average includes effects as low as zero and as high as a +13 improvement 

index (table 1). Unfortunately, the studies did not examine the effects of various components of summer 
counseling, and there are too few studies to draw conclusions about why the program may have had an 
effect in some studies but not in others. Therefore, although positive on average, the evidence for the effect 
of summer counseling on postsecondary enrollment should be interpreted with caution. 


Finally, one study of summer counseling found a statistically significant positive effect of the intervention on 
student persistence in college into the fall and spring of students’ second year (Castleman et al., 2014). The 
improvement index for this outcome was +6, indicating that a summer counseling intervention can move 
the average student from the 50th to the 56th percentile in the distribution of students persisting in college 
through their second year (WWC, 2018). The extent of evidence for this outcome is small because it is based 
on only one rigorous evaluation (table 1). 


Summer bridge interventions showed potentially positive effects on postsecondary completion® 


Summer bridge programs aim to support students’ postsecondary transition by connecting students to 
social resources that can help them succeed in college. In contrast with summer counseling interventions, 
these programs are traditionally hosted by postsecondary institutions, introducing admitted students to 
the resources and requirements of the particular institution. This review identified one study of asummer 
bridge program, offered to all incoming students at a selective technical university in the southeastern 
United States (Murphy et al., 2010). In the five-week program, students took a short non-credit-bearing 
course in calculus, chemistry, computer science, or English composition designed to mimic a college-level 
course. Upperclassmen served as coaches to support the students. Although largely focused on academic 
skills, this intervention included a nonacademic component that integrated families by inviting them 

to family-support sessions and award sessions. The family-support sessions included discussions with 
students and their families about housing arrangements, food plans, and coed visitation to educate both 
students and families about common pitfalls in the college experience (Murphy et al., 2010). The Murphy 
et al. study (2010) demonstrated an effect of the summer bridge program on college completion with an 
improvement index of +4, which corresponds to moving the average student from the 50th to the 54th 
percentile in the distribution of students completing college as a result of the summer bridge program. 
However, with only one study, the extent of evidence for the effect of summer bridge programs on 
postsecondary completion is small (WWC, 2016a). 


5 In the WWC systematic review of summer bridge interventions, only one study met WWC standards with or without reservations (WWC, 
2016a). This study demonstrated a positive effect on postsecondary completion (Murphy et al., 2010). The WWC characterized summer bridge 
interventions as having potentially positive effects on postsecondary completion. 
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Summary of the evidence for nonacademic interventions on the overall study population 


This systematic review identified five interventions with well-designed and well-implemented studies 

that showed positive effects on students’ postsecondary enrollment, persistence, or completion in the 
overall study population. FAFSA interventions showed the most promise overall, with positive effects 

on postsecondary enrollment and persistence, while FLIGHT, college counseling, summer counseling, 

and summer bridge programs showed positive effects on only one outcome each (postsecondary 
enrollment for FLIGHT and college counseling, persistence for summer counseling, and completion for 
summer bridge programs). Additionally, the extent of evidence for the effects of FAFSA interventions on 
postsecondary enrollment is medium to large, reflecting two studies that met WWC standards. The extent 
of evidence for each of the other findings is small. 


Table 1. Summary of the findings on nonacademic interventions for the overall study population, including 
the effectiveness ratings, average improvement index, and extent of evidence across studies for each 
intervention, by outcome, 2006-2019 


Outcomes 


Intervention Rating Postsecondary Postsecondary Postsecondary 
across studies enrollment persistence completion 
Number of studies 2 1 0 
FAESA Effectiveness rating @ Potentially positive Potentially positive na 
Avg improvement index ° +8 +9 na 
Extent of evidence ‘ Medium to large Small na 
Number of studies 1 0 0 
Effectiveness rating Potentially positive na na 
FLIGHT ‘ 
Avg improvement index +39 na na 
Extent of evidence Small na na 
Number of studies 1 1 0 
: Effectiveness rating Potentially positive Indeterminate na 
College counseling : : 
Avg improvement index +5 na na 
Extent of evidence Small Small na 
Number of studies 5 1 0 
___ Effectiveness rating Mixed Potentially positive na 
Summer counseling : : 
Avg improvement index +5 +6 na 
Extent of evidence Medium to Large Small na 
Number of studies 0 0 1 
Summer bridge Effectiveness rating na na Potentially positive 
program Avg improvement index na na +4 
Extent of evidence na na Small 


a. The effectiveness rating summarizes the direction and statistical significance of effects found across studies. 
b. The average improvement index averages the effect of an intervention across studies of the same intervention and expresses the average effect 
in percentile units. For binary outcomes, such as college completion, an improvement index of +1 corresponds to moving the average student 


from the 50th to the 51st percentile in the distribution of students completing college as a result of the intervention. 


c. The extent of evidence reflects the number of studies, schools, and students included in the analyses that contributed to the evidence for an 
intervention. “Small” indicates that the evidence is based on only one study, one school, or fewer than 350 students. “Medium to large” indicates 
that the evidence is based on at least two studies, in two schools, and with more than 350 students. 


Note: FAFSA is Free Application for Federal Student Aid. FLIGHT is Facilitating Long-Term Improvements in Graduation and Higher Education for 
Tomorrow. na is not applicable, indicating that no well-designed and well-implemented studies investigated the effect of the intervention on this 
outcome. No studies investigated the effect of any of these interventions on postsecondary academic performance. 


Source: Authors’ review of the What Works Clearinghouse intervention reports and single-study reviews. 
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Upward Bound was the only program that showed a positive impact on 
postsecondary enrollment in projects taking place in rural locales that 
served high-poverty populations. 


Upward Bound is a federal program that provides high school students from low-income households and first- 
generation college students with academic and nonacademic supports, such as cultural activities, college tours, 
and help with FAFSA completion, all with the aim of increasing postsecondary enrollment and completion. 
Arandomized controlled trial of Upward Bound found no overall impact of the program on any of the 
postsecondary outcomes of interest for this report (Myers et al., 2004). A follow-up study disaggregated results 
by a series of student-, project-, and school-level variables, including location of the project (rural or urban) 
(Seftor et al., 2009) and found a positive impact of Upward Bound on students’ postsecondary enrollment in 
projects taking place in rural locales. Additionally, during the time frame of the Upward Bound studies (1992 to 
1994), projects were required to enroll at least two-thirds low-income students. Thus, the impact on students 
in rural locales includes high-poverty students as well (Seftor et al., 2009). However, it should be noted that 
although the study was published within the eligible time frame for this review, the intervention took place 
between 1992 and 1994. The program has changed substantially since that time, and thus the findings of the 
study may not be generalizable to current conditions. 


None of the studies with overall impacts discussed above disaggregated the effects of their interventions 
on postsecondary outcomes for rural populations. The FLIGHT study included three rural counties, but the 
study did not report the effect of FLIGHT on postsecondary outcomes for the rural populations separately 
from the urban populations (Philp, 2015).° 


Evidence suggests summer counseling, FLIGHT, and Upward Bound might 
be effective with high-poverty populations. 


Castleman et al.'s study (2014) of summer counseling examined outcomes for postsecondary enrollment 
and persistence separately for high-poverty populations in two sites. In one site, Boston, summer 
counseling showed positive effects for students from high-poverty populations on enrollment and 
persistence into their sophomore year of college. The improvement index for this effect was +14, which 
corresponds to high-poverty students moving from the 50th to the 64th percentile in the distribution of 
students persisting into their sophomore year of college as a result of summer counseling (WWC, 2018). 
Additionally, the FLIGHT program specifically seeks to engage students who qualify for free or reduced- 
price lunch, and the one study of FLIGHT that met the WWC design standards found positive results 
among the participants (Philp, 2015). Likewise, as discussed above, the Upward Bound program included 
a large proportion of high-poverty students, and thus its impacts on rural students included high-poverty 
students as well (Seftor et al., 2009). 


Although FAFSA interventions showed the most promise overall, their effects on high-poverty populations 
are untested. Both studies of FAFSA interventions included high-poverty populations in the sample but 
did not disaggregate results specifically for those populations. 


° The author disaggregated high school outcomes but not postsecondary outcomes for rural populations. 
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Overall, summer counseling and FLIGHT showed the most promise for K-12 stakeholders with a high 
proportion of students in poverty, as both have been shown to have potentially positive effects on 
persistence in postsecondary education, specifically for a subpopulation of high-poverty students, and 
in studies that took place within the last 10 years. Nonetheless, the extent of evidence for each is small 
because the number of studies that met design standards is limited. 


The information available on the effects of nonacademic interventions on 
postsecondary enrollment and success, particularly for rural and high- 
poverty populations, may have significant gaps. 


The results of the literature search and screening process indicate large gaps may exist in the research 

that could help schools, districts, and states select interventions that meet the Every Student Succeeds 

Act's evidence requirements and are likely to benefit their rural and high-poverty populations. These 
possible research gaps fall into six main categories. First, although the search returned a large number of 
studies that examined nonacademic competencies, more than half of these were excluded because they 
did not focus on an intervention (1,033 studies, 58 percent, table 2). Instead, these studies focused on 

the nonacademic competencies themselves, naturally occurring or preexisting student characteristics, or 
family or demographic background variables. Second, 29 percent of the studies identified in the search 
were excluded because they focused on populations older than 19 or younger than 5. This included one 
preschool intervention, a large number of college-level interventions (including 36 that took place in the first 
year of postsecondary education), and interventions designed for adults who were not in school. Third, 11 
percent of the studies were excluded because they did not examine postsecondary outcomes. These studies 
often focused on the effects of nonacademic interventions on students’ high school outcomes, such as 
course grades and high school graduation. 


Table 2. Of the 1,777 studies researchers found in their search of ProQuest’'s Education Database and the 
What Works Clearinghouse website, nearly two-thirds did not examine a nonacademic competency, and 
more than half did not examine an intervention 


Reason for Exclusion Number of studies Percent of studies 
Did not examine a nonacademic competency 1,159 65.2 
Did not examine an intervention 1,033 58.1 
Did not focus on students ages 5 to 19 514 28.9 
Did not include the postsecondary outcomes of interest 201 11.3 
Total number of studies screened 1,777 100 


Note: Categories are not mutually exclusive; a study could be excluded from the review for more than one reason. 


Source: Authors’ review of the evidence and What Works Clearinghouse intervention reports. 
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The fourth gap in research was in the quality of the study designs. Of the 65 studies that examined 
nonacademic interventions with the relevant population and outcomes, only 17 used well-designed and 
well-implemented experimental or quasi-experimental designs (see figure 1). Fifth, the results of this 
evidence review show a gap in the disaggregation of research results. Only three studies that met the 
criteria for inclusion in this review examined effects separately for high-poverty populations and only 
one examined effects explicitly for rural populations. Additionally, the one study that examined a rural 
population was implemented between 1992 and 1994, and the program has changed substantively since 
then (Seftor et al., 2009). 


Finally, among the 17 well-designed and well-implemented studies that met the inclusion standards 

for this review, only eight found statistically significant positive effects for the outcomes of interest. 

The remaining nine studies found indeterminate effects (eight studies) or did not provide adequate 
information to calculate effect sizes (one study, Nathan, 2013). In addition to the three studies that found 
indeterminate effects of summer counseling (discussed above), the studies with indeterminate effects 
examined developmental summer bridge programs (Barnett et al., 2012), EXCEL'® (Bergin et al., 2007), and 
the Quantum Opportunity Program (Rodriguez-Planas, 2010; Schirm et al., 2006). This review found no 
additional well-designed, well-implemented studies with positive, statistically significant effects of these 
three programs on the outcomes of interest for the overall study population or for rural or high-poverty 
populations in particular. Therefore, despite existing randomized controlled trials or quasi-experimental 
studies of these three interventions, they lack strong evidence of their effect on postsecondary outcomes. 


10 Despite being spelled in all capital letters, it does not appear that EXCEL is an acronym. In the Bergin et al. (2010) study, the intervention is 
always referred to only as “EXCEL.” 
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Implications 


Nationwide, 19 percent of U.S. public school students are enrolled in a rural school (Lavalley, 2018), 
and 24 percent are enrolled in a high-poverty school (NCES, 2019c). These students face challenges 
that are different from those faced by their nonrural and low-poverty peers. One example is in building 
the nonacademic competencies necessary for postsecondary enrollment and success. To address 
these challenges, decisionmakers need evidence for nonacademic interventions designed to increase 
postsecondary enrollment and success, particularly among rural and high-poverty populations. This 
review shows that the only intervention with evidence of impact in rural communities serving high- 
poverty students is Upward Bound as implemented in 1992-1994. 


This indicates there are likely gaps in evidence that researchers can address in five ways. First, researchers 
can carry out more studies on nonacademic competencies and the link between these competencies 

and postsecondary outcomes. A substantial portion of the research on postsecondary enrollment and 
success is focused on academic, rather than nonacademic, competencies. The screening process for this 
review eliminated 65 percent of the studies returned from the searches because they did not address 
nonacademic competencies. Nonacademic competencies have been linked to postsecondary enrollment 
and success (Conley, 2007; Duckworth et al., 2007; Kinsantas et al., 2008; McDonough, 1997; Nagaoka et 
al., 2009), particularly for rural and high-poverty populations (Terenzini et al., 2001), but more research is 
needed to understand how these competencies promote postsecondary enrollment and success. 


Second, in addition to more research on nonacademic competencies, researchers can also carry out more 
studies on specific interventions designed to improve these competencies. This review found that the 
research that does exist on nonacademic competencies often focuses on the competencies themselves, 
rather than on interventions designed to develop these competencies. Investigation into nonacademic 
competencies is an important first step in designing interventions that aim to improve the nonacademic 
knowledge, skills, and behaviors that have been shown to affect postsecondary outcomes. This type of 
research can help the field understand how nonacademic competencies develop and which have the 
strongest relationship with student outcomes. With this information, intervention designers can create 
programs that are hypothesized to develop these competencies and thus impact student outcomes. 
However, decisionmakers rarely design their own programs or interventions from scratch; they need to 
be able to identify programs and policies that are already designed and within their power to implement. 
Researchers need to provide evidence on whether specific nonacademic interventions have the 
hypothesized effects on students. This will enable stakeholders to take action in implementing successful 
interventions in their schools and districts. 


Third, researchers can carry out studies of nonacademic interventions that take place in secondary 
schools or with students ages 5 to 19. A substantial portion of the research on the effects of nonacademic 
interventions on postsecondary outcomes took place entirely in postsecondary educational settings. 

This review excluded more than one-quarter of the studies identified in the initial search because they 
focused on students older than 19 or younger than 5, or took place entirely with postsecondary students 
already enrolled in postsecondary institutions (for example, first-year experience courses). Although these 
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interventions are important to investigate, the studies will not yield information that will help those in the 
K-12 system effectively prepare high school graduates for success in postsecondary education. Increasing 
the available evidence about nonacademic interventions currently in place in K-12 schools and their effect on 
postsecondary outcomes can help fill this gap and potentially benefit the K-12 and higher education systems. 


Without additional research, schools and districts are implementing programs with limited evidence of 
effectiveness. For example, mentoring programs are designed to put prospective college students in 
touch with adults who know the ins and outs of the college application and enrollment process and have 
firsthand knowledge of how to succeed in a postsecondary institution. One program with a mentoring 
component (FLIGHT) showed positive effects on postsecondary enrollment, specifically with a high- 
poverty population. However, the body of rigorous evidence for mentoring is small, reflecting just this 
one study. Many other mentoring programs either have not been studied or have not shown effects on 
postsecondary outcomes. For example, the Quantum Opportunity Program (QOP) provides mentors to 
students as part of a long-term after-school program. However, WWC reviews of QOP studies showed 
indeterminate effects of the program on postsecondary persistence (WWC, 2007, 2016b). While two 
studies met WWC standards with reservations (Rodriguez-Planas, 2010; Schirm et al., 2006), neither 
showed statistically significant effects on the postsecondary outcomes of interest. Thus, the field needs 
more research on mentoring interventions to determine whether they are effective for improving 
students’ postsecondary outcomes before they are implemented more broadly. 


Fourth, researchers should address the gap in the evidence for rural and high-poverty populations. 
Decisionmakers are increasingly looking to identify interventions that are most likely to benefit students 

in their particular context, a practice embedded in the Every Student Succeeds Act. Rural and high- 

poverty schools and districts have unique strengths and challenges, and the lack of information on how 
interventions perform in those contexts presents a dilemma for decisionmakers. Without additional 
research, schools and districts serving rural and high-poverty populations may implement interventions 
that are not designed for their circumstances and may not achieve intended outcomes. The consequences 
are wasted resources and inadequately supported students. Researchers can address this gap in two ways: 
(a) they can carry out more studies specifically focused on rural and high-poverty areas; (b) researchers 
using large national datasets or multi-site studies could consider rural and high-poverty populations in their 
sampling and disaggregate their results for high-poverty and rural populations. Many studies reviewed here 
included high-poverty populations, but only three disaggregated results for these populations. 


Fifth, researchers should address the gap in both the quality of the research and in the quality of the 
reporting of research in this field. Although 65 studies returned by the search of ProQuest’s Education 
Database and the WWC website were focused on the populations, interventions, and outcomes of 
interest, and used designs appropriate to be included in this review, only 17 met WWC standards with or 
without reservations (see figure 1). The remaining 48 studies did not meet standards, most commonly (45 
of the 48) because the authors failed to demonstrate that the treatment and comparison groups were 
equivalent before the intervention began. This can happen for two reasons. On the one hand, the study 
may not have been well implemented, such that the treatment and comparison groups were substantially 
different, even before the intervention began. In this case, any effects attributed to the treatment may 
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in fact be due to preexisting differences. Researchers can address this issue through more frequent 

use of random assignment, greater attention to attrition, and more careful selection of comparison 
group participants in non-random-assignment studies. Another way a study may fail to demonstrate the 
equivalence of the treatment and comparison groups at baseline is if it did not report data at baseline. 
In these cases, the groups may not have been substantially different, but the study authors neglected 

to provide evidence of that fact. This may lead to underreporting of significant results. Researchers can 
improve the quality of evidence in the field by ensuring that they report information on the baseline 
differences between treatment and comparison groups. 


Additionally, only three studies that met WWC standards reported results separately for high-poverty 
populations, and only one disaggregated for rural populations. While the failure to disaggregate for rural 
students may reflect an inadequate number of research studies being conducted in rural communities, 
approximately one-third of the studies focused on nonacademic interventions with postsecondary 
outcomes reported the proportion of students in poverty.'' This indicates that the researchers had the 
ability to disaggregate results for students living in poverty, if not the statistical power to detect effects for 
subgroups. 


This insufficient reporting of important aspects of the study design and of the subgroups of students 
included in the analysis may lead to underreporting of the number of effective interventions for rural and 
high-poverty populations. As a result, stakeholders may choose not to implement interventions that have 
the potential for positive effects. 


Finally, this review was unable to identify which specific program components led to improved 
postsecondary outcomes. All six interventions discussed in this review include multiple components, 
sometimes including academic elements (FLIGHT, Upward Bound, and summer bridge). However, none of 
the studies examined the separate effect of each intervention component, and none of the interventions 
had enough studies or enough variation in implementation for this review to attempt to tease apart 

the effects. Without substantially more research, it is not possible to determine which piece of each 
intervention is most responsible for the outcomes, or if all the elements work together as a whole. 

Each school or district has its own set of resources and constraints, and local actors are rarely able to 
implement interventions exactly as designed. Additional information on the critical components of an 
intervention and their connection to the desired outcomes can support stakeholders in implementing a 
program that leverages their existing resources and still has an effect on postsecondary outcomes. 


"' This includes studies that were ultimately excluded from the review because they did not meet WWC standards or did not have positive effects. 
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il Limitations 


This review is limited by insufficient data reporting in the majority of well-designed studies addressing the 
populations and outcomes of interest. As discussed above, 45 of the 65 studies that were focused on the 
populations, interventions, and outcomes of interest, and that used designs appropriate to be included in 
this review, did not meet WWC standards because they failed to show that the treatment and comparison 
groups were similar before the intervention began.'* As a result, the number of interventions with 
positive effects may be underreported. Additionally, insufficient data reporting prevented the authors 

of this review from systematically analyzing the similarities and differences in the components of these 
nonacademic interventions and their effects on students. 


This review may also be limited by the approach to searching for and including literature. The research 
team designed the approach to be systematic and replicable. However, searching only an online database 
and WWC's website may have excluded studies that did not include the selected keywords or were not 
housed in ProQuest's Education Database. Additionally, the particular key words and phrases used as 
search terms may not have uncovered all relevant studies in ProQuest’s Education Database. As a result, 
relevant studies may have been omitted from this review. 


Similarly, the review was limited to studies published between 1998 and 2018. This may have eliminated 
well-designed and well-implemented experimental or quasi-experimental studies published more than 

20 years ago or in the time between the completion of the search and the publication of this study. The 
authors chose to limit the date range to increase the chances of finding studies that reported the full set 
of data required by WWC and of reaching authors to gather additional information not reported in the 
studies. Given the challenges of identifying studies that included all the needed data discussed above, it is 
unlikely that extending the time frame backward would yield much additional evidence. 


'2 The Regional Educational Laboratory Appalachia researchers queried the authors of the studies that ProQuest returned which had not already 
been reviewed by the WWC (four studies). These queries followed WWC procedures, but no authors were able to provide the information 
necessary to establish baseline equivalence. Three authors did not reply to multiple attempts, and one author replied but was unable to provide 
the information. 
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Appendix A: Nonacademic competencies 


Research has used a variety of terms for the nonacademic competencies discussed in this report. 
Likewise, studies vary in how they define what the category includes and excludes. This review draws 

its definition of nonacademic competencies largely from frameworks for college readiness developed 

by Barnett (2016) and Conley (2012). In addition to the academic skills necessary for success in 
postsecondary education, both frameworks emphasize what Barnett calls “noncognitive skills,” and 

what Conley refers to as “key learning skills and techniques.” These skills include students’ ability to set 
personal goals, their internal motivation and self-regulation, and their persistence in pursuing long-term 
goals. These skills have been linked to postsecondary enrollment and success (Duckworth et al., 2007; 
Kinsantas et al., 2008), and Barnett identifies at least one example of schools working to increase these 
skills in their students (Barnett, 2016). Additionally, there is evidence that the relationship between these 
skills and postsecondary outcomes may be particularly strong for students from low-income backgrounds 
(Terenzini et al., 2001). 


Besides key learning skills and techniques, both frameworks also discuss the role of cultural capital or 
“key transition knowledge and skills” (Conley, 2012). Competencies in this area include an understanding 
of the norms or expectations of college, understanding the application and financial aid process, and 
the ability to select a college appropriate for one’s interests, skills, and needs. As with noncognitive 
skills, research shows that cultural capital impacts postsecondary outcomes, especially for traditionally 
underrepresented students (Conley, 2007; McDonough, 1997; Nagaoka et al., 2009). 


In its inclusion criteria, this review defines “nonacademic competencies” as both the key learning skills 
and techniques and the key transition knowledge and skills outlined by Barnett (2016) and Conley (2012). 
Thus, studies that examined other factors that are not explicitly academic, such as race, socioeconomic 
status, and institutional barriers to success, were excluded as not addressing nonacademic competencies. 
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Appendix B: Methods 


At the start of the project, researchers performed a series of searches of ProQuest’s Education Database 
using variations on a set of broad search terms (nonacademic, noncognitive, out-of-school time, social- 
emotional, rural college access and success, and high-poverty college access and success) and the 
outcomes of interest (postsecondary enrollment, postsecondary performance, postsecondary persistence, 
and postsecondary completion), including or excluding quotation marks, Boolean operators, and 
wildcards’? to narrow or broaden the search results. The goal of this step was to obtain the broadest 
possible list of relevant studies, while excluding studies not relevant to the review. At this first stage, the 
research team ruled out using the outcomes as search terms, as they returned substantially fewer studies 
than did searches excluding those terms. To ensure that no studies were inadvertently excluded, the team 
instead added the word “outcome” to many of the searches. The final search terms and search results are 
in table B1. 


Once the search terms were finalized, one principal investigator trained five additional researchers in the 
searching and screening process. The training covered the process for accessing and searching ProQuest's 
Education Database, saving the results of searches, completing the screening form, and reviewing the 
operational definitions for the inclusion criteria (see box 1 in the body of the report). The principal 
investigator demonstrated screening five studies from the “nonacademic” search term. Following this initial 
training, each researcher screened the next 10 studies from the “nonacademic” search term. At a follow- 

up meeting, the team discussed discrepancies in their responses to the screening items and updated a 
frequently asked questions (FAQ) document with additional information on the screening criteria. Each 
researcher double-screened a minimum of 10 more studies, continuing until reaching 80 percent agreement 
with the principal investigator across five screening questions. Once each researcher had reached 

reliability in the screening process, he or she was assigned a Search term or terms and began to screen 
independently. Screeners completed an online checklist to document the reasons for inclusion or exclusion 
(see appendix C for a print version of the online screening form). The full set of search terms, as well as 

the Boolean operators, wildcards, and variations used in each case, are shown in table B1. Many studies 
appeared in more than one search. For the first four months of screening, this overlap was used to continue 
to monitor reliability between screeners, and screening pairs met to discuss and resolve discrepancies in 
their responses to the checklist. After that point, researchers screened all studies only once. 


'3 Boolean operators are words such as “and,” “or,” and “not” that join search terms. Wildcards are asterisks added to search terms to allow the 
search engine to complete the word. For example, “skill*” would return studies that include “skill,” “skills,” or “skilled” in their description. 
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Table B1. Search terms and results 


Broad search Number of studies 
term returned 


Specific search terms used in ProQuest’s Education Database 


("nonacademic" OR "non-academic") AND ("postsecondary outcome*" OR 


Monacademie "post-secondary outcome*" OR "college outcome*") AND intervention* aot 
Ww ith W Ww i itt Ww Ww kN 

Noncognitive ( noncognitive" OR "non eee ) AND ( pastsscondary outcome*" OR 540 
post-secondary outcome*" OR "college outcome*") 

Out-of-school ("out of school time" OR "out-of-school time") AND ("postsecondary 39 

time outcome*" OR "post-secondary outcome*" OR "college outcome*") 

Sadia: ("social emotional skill*" OR "social-emotional skill*" OR "socio emotional 

ai aaeaedl skill*" OR "socio-emotional skill*") AND ("postsecondary outcome*" OR "post- 30 


secondary outcome*" OR "college outcome*") 


Rural college rural AND ("college access") AND ("postsecondary outcome*" OR "post- 


access secondary outcome*" OR "college outcome*") a8 
Rural college rural AND ("college success") AND ("postsecondary outcome*" OR "post- 435 
success secondary outcome*" OR "college outcome*") 
. ("high poverty" OR "high-poverty") AND ("college access") AND 
High-poverty W NN W *N W 
("postsecondary outcome*" OR "post-secondary outcome*" OR "college 121 
college access é 
outcome*") 
Histnaver ("high poverty" OR "high-poverty") AND ("college success") AND 
BrP y ("postsecondary outcome*" OR "post-secondary outcome*" OR "college 126 


college success 
g outcome*") 


Note: Categories are not mutually exclusive. Studies were often returned in more than one search. 


Source: Authors’ search of ProQuest’s Education Database between June and December 2018. 


In addition to searching ProQuest's Education Database, the two principal investigators also examined 
the What Works Clearinghouse (WWC) list of intervention reports and reviews of individual studies in 

the “Path to Graduation” and “Postsecondary” topic areas. In compiling its intervention reports, the 

WWC carries out a full literature search for all studies examining a particular intervention. The principal 
investigators first created a list of all the studies included in each intervention report or study review that 
reported on postsecondary outcomes. Then, one principal investigator screened these studies using the 
same inclusion criteria used on the studies found in ProQuest's Education Database. 


After this initial screening, the study team compiled the list of studies that met the inclusion criteria, 
and researchers screened each study for design criteria. Only researchers with advanced degrees in 
quantitative research methods and design or who were WWC-certified carried out the second-stage 
screening. At this stage, researchers excluded studies that used qualitative designs; used quantitative 
descriptive designs (that is, did not perform statistical tests of the differences between groups); used 


Nonacademic Interventions for Postsecondary Enrollment and Success with Rural and High-Poverty Populations: 27 
A Systematic Evidence Review 


quantitative designs with no comparison group or regression discontinuity designs;'* or had already 
been reviewed by the WWC (for example, those found on the WWC website). Studies that had already 
been reviewed by the WWC were included in this review if they met the inclusion criteria, but they were 
not reviewed again using the WWC standards. Studies that had not yet been reviewed by the WWC were 
subject to a WWC review. At this stage, researchers documented the basic research design and any 
reasons for exclusion in an online spreadsheet. 


Once the list of studies had been narrowed to those that examined nonacademic interventions with the 
relevant population and outcomes, had designs eligible to be reviewed using WWC standards, and had 
not already been reviewed by the WWC, the study team carried out a WWC review of each eligible study, 
including author queries when needed and peer review of the ratings. The researchers involved in this 
step were all certified WWC reviewers under version 3.0 of the WWC standards. 


The study team found a total of six studies in ProQuest’s Education Database that met the criteria 

for inclusion in the report and had not been reviewed by the WWC. For the WWC reviews by Regional 
Educational Laboratory Appalachia (REL AP), each of these six studies was assigned to two certified 
reviewers. Each reviewer completed a version 3.0 WWC Study Review Guide provided by the WWC. In this 
form, reviewers documented the study design (randomized controlled trial, quasi-experimental design, 

or cluster randomized controlled trial), outcomes, potential confounds, attrition, baseline equivalence, 
and an overall rating (see box 1 in the body of the report). For studies that met standards with or 

without reservations, reviewers also noted additional study details, such as the setting, sample sizes, and 
comparison condition. After both completed the form, a master reviewer compared the two responses. In 
cases of discrepancy, the two reviewers met with the master reviewer to confer. The master reviewer then 
completed a master review form documenting the final ratings for the study. 


Of the six studies eligible for the REL AP WWC review, four had insufficient information to make an overall 
rating. For these studies, the research team completed author queries. Researchers contacted the author 
of each study, outlining the specific information that was missing and necessary to complete the review. 
Authors had three weeks to respond to a query, but researchers extended this deadline because few 
authors responded. Researchers continued to reach out to authors for four months to attempt to obtain 
the information necessary to complete a WWC review. For a variety of reasons, none of the authors was 
able to supply the necessary information. Three never replied to the queries, and one explained he no 
longer had access to the original data used for the study. Most of the studies in this category (three of the 
four) were Ph.D. dissertations, and authors were rarely still at the institution where the study was carried 
out; most were no longer in education research. 


The two studies with sufficient information to complete a WWC review were sent to the WWC, along with 
the master review form, to be double-coded in an official WWC review. The WWC gave feedback on the 
master Study Review Guide, and the researchers made adjustments as needed. Of these two studies, one 
did not meet WWC standards and one met WWC standards without reservations but did not include the 


'4 Regression discontinuity designs were not eligible to meet WWC standards under the version 3.0 standards. Thus, studies using this approach 
were excluded from this review. 
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information necessary to calculate effect sizes. See appendix D for the full list of studies that examined 
nonacademic interventions with the relevant population and outcomes. 


After gathering the full set of interventions that met the screening criteria and completing the WWC 
reviews, researchers determined that several interventions that met the student-age criterion (for 
example, first-year experience courses and Linked Learning communities) were nonetheless not relevant 
for this review. The initial reason for extending the age range in the screening criteria to 19 was to include 
programs focusing on over-age high school students and those taking place in the summer after high 
school. However, interventions implemented entirely with postsecondary students already enrolled in 
postsecondary institutions are impossible for K-12 stakeholders to implement in their schools. Thus, 
although these studies technically met the initial inclusion criteria, they were excluded from the review. 


Finally, using the list of studies that met all inclusion criteria, the principal investigators identified 
nonacademic interventions that had at least one well-designed and well-implemented study that included 
the relevant population and found positive effects on at least one outcome of interest. 
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Appendix C: Screening form 


This appendix includes a print version of the online form used in the literature screen. 


Reviewer Information 


Reviewer name: 


Search term category (select one): 
O Nonacademic 

O Noncognitive 

O Out-of-school time 

O Social-emotional 

O Rural college access 

O Rural college success 

O High-poverty college access 

O High-poverty college success 


Study citation: 


Study type 


Is the study any of the following? 


Yes iN To) Don’t know 
Literature review Oo Oo Oo 
Meta-analysis Oo Oo Oo 
Study summarizing other research Oo Oo Oo 
News article Oo Oo Oo 
Promotional material for an intervention ©) O Oo 
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Participants 


Do the study participants include K-12 students or students approximately ages 5-19 AT THE TIME OF THE 
INTERVENTION? (interventions that students receive the summer after high school graduation are also 
included) 


Yes No 
O Oo 
Focus 


Does the study focus on an INTERVENTION? 


Yes No 
Oo Oo 


Does the study investigate a NONACADEMIC component, activity or factor? 


Yes No 
O O 
Outcomes 


Does the study include any of the following outcomes? 
(Select all that apply) 

O Postsecondary enrollment or matriculation 

O Postsecondary educational persistence or retention 
O Postsecondary academic performance 

O Postsecondary completion or graduation 


O None of these 
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Appendix D: Full list of studies that examined 
nonacademic interventions with the relevant 
population and outcomes 


This appendix lists all studies that met the study's first six inclusion criteria: 


1.Included an original research study 
2.Tested the effects of an intervention or policy 


3. Included participants who were ages 5 to 19 years at the time of the intervention, excluding any 
interventions that took place entirely with postsecondary students at postsecondary institutions 


4.Examined an intervention with at least one nonacademic component 


5. Measured one of the four outcomes of interest: postsecondary enrollment; postsecondary 
persistence; postsecondary academic performance; and postsecondary completion 


6.Was published in English between January 1, 1998, and December 20, 2018 (the date the searches 
were completed) 


Table D1 shows only those studies that met the above six criteria and also met What Works 
Clearinghouse (WWC) standards and showed statistically significant, positive results. Tables D2 
through D3 display the studies that met the above inclusion criteria but were excluded from analysis in 
this report, by the reasons for their exclusion. 
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Table D1. Studies eligible for inclusion in the report by What Works Clearinghouse (WWC) rating 


Full citation Reviewed by 

Met WWC Group Design Standards without Reservations the WWC? Intervention 

Carrell, S., & Sacerdote, B. (2013). Why do college going interventions work? (NBER vee College 
Working Paper 19031). Cambridge, MA: National Bureau of Economic Research. counseling 

Castleman, B. L., Page, L. C., & Schooley, K. (2014). The forgotten summer: Does 
the offer of college counseling after high school mitigate summer melt among Vee Summer 
college-intending, low-income high school graduates? Journal of Policy Analysis counseling 
and Management, 33(2), 320-344. https://eric.ed.gov/?id=EJ1027721 

Murphy, T. E., Gaughan, M., Hume, R., & Moore, S. G. (2010). College graduation rates namics 
for minority students in a selective technical university: Will participation ina summer , 

; : ; ; ; ; Yes bridge 
bridge program contribute to success? Educational Evaluation and Policy Analysis, 32(1), 
program 
70-83. 

Myers, D., Olsen, R., Seftor, N., Young, J., & Tuttle, C. (2004). The impacts of regular awed 
Upward Bound: Results from the third follow-up data collection. Princeton, NJ: Yes ae . 
Mathematica Policy Research. 

Page, Lindsay C., Castleman, Benjamin, & Meyer, Katharine. (2018, May 14). 

Customized nudging to improve FAFSA completion and income verification. Available at Ves FAFSA 
SSRN (formerly Social Science Research Network, online): 
https://ssrn.com/abstract=2854345 or http://dx.doi.org/10.2139/ssrn.2854345 
Seftor, N. S., Mamun, A., & Schirm, A. (2009). The impacts of regular Upward Bound 
F : Upward 
on postsecondary outcomes 7-9 years after scheduled high school graduation. Yes Baud 


Princeton, NJ: Mathematica Policy Research. 


Met WWC Group Design Standards with Reservations 


Bettinger, E., Long, B., Oreopoulos, P., & Sanbonmatsu, L. (2012). The role of 
application assistance and information in college decisions: Results from the H&R Yes FAFSA 
Block FAFSA experiment. Quarterly Journal of Economics, 127(3), 1205-1242. 


Philp, J. D. (2015, October). FLIGHT: Final evaluation report. Columbia, SC: The 
Evaluation Group. 


Yes FLIGHT 


Note: FAFSA is Free Application for Federal Student Aid. FLIGHT is Facilitating Long-Term Improvements in Graduation and Higher Education 
for Tomorrow. WWC is What Works Clearinghouse. Studies cited in the text are also included in the report references section. Studies that were 
screened but ultimately excluded and not mentioned specifically in the report are not in the report references section. 


Source: Authors’ search of ProQuest’s Education Database and the WWC website between June and December 2018 and WWC ratings in January- 
June 2019. The WWC confirmed ratings in July-October 2019. 
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Table D2. Studies that met the initial eligibility criteria but were excluded from the report because they did 
not meet What Works Clearinghouse (WWC) standards 


Reason for not 
meeting WWC 
standards 


Reviewed 
by the 
WWC? 


Ul | Medit-litey a) 


Intervention 


Allen, L. (2001). An evaluation of the University of Missouri- 


Baseline 


Rolla minority engineering program 7-week summer bridge : ' Summer 
; ; f ‘ ‘ : equivalence is : 
program. Doctoral dissertation, University of Missouri- Yes bridge 
: : ‘ : necessary and not 
Columbia, Rolla, MO. Available from ProQuest Dissertations actablished program 
and Theses database (UMI No. 3012945). 
Appenzeller, E. A. (1998). Transition to college: An assessment of , 
: ; Baseline 
the adjustment process for at-risk college students. Doctoral saliivalaneels Summer 
dissertation, Claremont Graduate University and San Diego Yes q bridge 
seaapee . ; necessary and not 
State University, Claremont and San Diego, CA. Available from aeeabliched program 
ProQuest Dissertations and Theses database (UMI No. 9821495). 
Cabrera, N. L., Miner, D. D., & Milem, J. F. (2013). Can a Baseline 
’ 5 . ; ; : Summer 
summer bridge program impact first-year persistence vee equivalence is bnidee 
and performance? A case study of the New Start Summer necessary and not a” ae 
Program. Research in Higher Education, 54(5), 481-498. established Pros 
Chism, L. P., Baker, S. S., Hansen, M. J., & Williams, G. (2008). Baseline 
‘ ‘ . : . Summer 
Implementation of first-year seminars, the Summer Academy Was equivalence is Kids 
Bridge Program, and themed learning communities. necessary and not “6 an 
Metropolitan Universities, 19(2), 8-17. established Pros 
Citty, J. M. (2011). Factors that contribute to success ina Baseline . Summer 
fi : ; : equivalence is : 
irst year engineering summer bridge program. Doctoral Yes bridge 
f : ‘ : ‘ i : necessary and not 
dissertation, University of Florida, Gainesville, FL. ; program 
established 
Clayton, A. B. (2016). Assisting students in the college choice Bac 
: : aseline 
process: Three essays on the role and effectiveness of : F : 
‘i : : me det equivalence is Counseling / 
college advising professionals in public high schools. Doctoral No Raceeeanranainek sauisiri 
dissertation, North Carolina State University, Raleigh, NC. ener tet 8 
https://repository.lib.ncsu.edu/handle/1840.16/11038 
Constantine, J. M., Seftor, N. S., Sama Martin, E., Silva, T., & 
Myers, D. (2006). Study of the effect of the Talent Search WWC did not 
program on secondary and postsecondary outcomes in Florida, Vas provide additional Talent Search 
Indiana, and Texas: Final report from phase I! of the national information on the 
evaluation. Princeton, NJ: Mathematica Policy Research. reason 
https://files.eric.ed.gov/fulltext/ED493358.pdf 
, : : : Baseline 
Doerr, H. M., Arleback, J. B., & Costello Staniec, A. (2014). Design : : Summer 
. ; hae equivalence is : 
and effectiveness of modeling-based mathematics in a summer Yes necessary and not bridge 
bridge program. Journal of Engineering Education, 103(1), 92-114. established program 
Douglas, D., & Attewell, P. (2014). The bridge and the troll Baseline : Summer 
; : equivalence is ; 
underneath: Summer bridge programs and degree Yes bridge 
completion. American Journal of Education, 121(1), 87-109 neces say ane net program 
‘ : ' ‘ established 
Elwell, T., West, M., & Scott, L. (2011). State-funded mentoring . 
emia } Baseline 
program performance mixed: Take Stock and Best Buddies Saliivalancele 
do well. Report No. 11-11. Tallahassee, FL: The Florida Yes seat endlnde FLIGHT 
Legislature, Office of Program Policy Analysis & Government Hemera 
Accountability (OPPAGA). 
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Table D2. Studies that met the initial eligibility criteria but were excluded from the report because they did 
not meet What Works Clearinghouse (WWC) standards (continued) 


Reviewed Reason for not 
Full citation by the meeting WWC Intervention 
WwWwc? standards 
. was Baseline 
Evans, R. (1999). A comparison of success indicators for program ; ‘ Summer 
a : ; equivalence is ; 
and non-program participants in a community college summer Yes bridge 
bridge program for minority students. Visions, 2(2), 6-14 mecessaly snd ioe program 
‘ ’ : ‘ established 
Fester, R. (2010). Participation in precollege outreach programs Baseline 
and the transition from high school to college. Doctoral No equivalence is Summer 
dissertation, University of Pennsylvania, Philadelphia. necessary and not counseling 
https://repository.upenn.edu/dissertations/AAI3429166/ established 
Fletcher, S. L., Newell, D. C., Newton, L. D., & Anderson- Baceline 
Rowland, M. R. (2001, November). The WISE Summer Bridge ; : Summer 
; ; as ; equivalence is . 
Program: Assessing student attrition, retention, and program Yes bridge 
: nue ; necessary and not 
effectiveness. Paper presented at the Frontiers in Education : program 
established 
conference, Reno, NV. 
Fosnacht, K. J. (2011). Access is not enough: How pre-college ; 
: Baseline 
programs impact students after college entry. Doctoral , : 

; : ; equivalence is Summer 
dissertation, UCLA, Los Angeles. https://search.proquest.com/ No necaecaniand abe cauneelin 
openview/989db74c502c1a4df98309bf1843439a/1?pq-origsit sc ae 8 
e=gscholar&cbl=18750&diss=y 

Baseline 
Gancarz, C. P., Lowry, A. R., McIntyre, C. W., & Moss, R. W. (1998). ; , Summer 
; ; . equivalence is : 
Increasing enrollment by preparing underachievers for Yes bridge 
college. Journal of College Admission, 160, 6-13 REGS sat) anal iot program 
‘ / : ‘ established 
Grisé, P., Chen, Y-S, Drennan, A., & Fisher, T. H. (2007). Baseline 
Mentoring in Florida, 2004-2005. Report prepared for the Vee equivalence is FLIGHT 
Florida Mentoring Partnership. Tallahassee, FL: Florida State necessary and not 
University. established 
Gutierrez, T. E. (2007). The value of pre-freshman support : 
. : . Baseline 
systems: The impact of a summer bridge program at UNM. : , Developmental 
: ‘ . j ; equivalence is ; 
Doctoral dissertation, University of New Mexico, Albuquerque, Yes summer bridge 
; é : necessary and not 
NM. Available from ProQuest Dissertations and Theses aétablished program 
database (UMI No. 3273438). 
Hall, E. R. (2008). Minority student retention program: Student Baseline 
‘ ‘ : : Developmental 
achievement and success program at Anne Arundel Community equivalence is : 
; d f Yes summer bridge 
College. Doctoral dissertation. Available from ProQuest necessary and not ronan 
Dissertations and Theses database (UMI No. 3342828). established Prog 
Hamilton, J. M. (1994). First academic year progress of summer Measures of 
1993 high-risk students in the Fresh Start program as effectiveness Developmental 
compared to similar students who entered Gainesville College Yes cannot be summer bridge 
during the fall of 1990. Gainesville, GA: Gainesville College, attributed solely to program 
Office of Planning and Institutional Research. intervention 
Hanks, C. L., Wartes, D., Levine, R., Gonzalez, R., Fowell, S., & 
Owens, G. (2007). Introducing the geosciences to Alaska Baseline 
Natives via the Rural Alaska Honors Institute (RAHI). Journal No equivalence is Summer bridge 
of Geoscience Education, 55(6), 507-513. Retrieved October necessary and not program 
10, 2019, from http://www.nagt.org/files/nagt/jge/abstracts/ established 
introducing geosciences_alaska.pdf 
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Table D2. Studies that met the initial eligibility criteria but were excluded from the report because they did 
not meet What Works Clearinghouse (WWC) standards (continued) 


Reason for not 
meeting WWC 


Reviewed 
Taycyarccyayiceyal 


UL Meg tclaeya| by the 
WwWwc? standards 


Haugen, D. E. (2012). College transition programs for community 


Baseline 


college students. Doctoral dissertation, University of Nevada, Yau equivalence is Summer bridge 
Reno, NV. Available from ProQuest Dissertations and Theses necessary and not program 
database (UMI No. 3511972). established 
Haught, P. A. (1996, April). Impact of intervention on Baseline 
disadvantaged first year students who plan to major in health Wee equivalence is Summer bridge 
sciences. Paper presented at the annual meeting of the necessary and not program 
American Educational Research Association, New York. established 
Herndon, L. G. (2007). Evaluation of the summer bridges Bacall 
Peete : ; aseline 
minority scholarship program at a community college. Sarieealcente Developmental 
Doctoral dissertation, Nova Southeastern University, Broward Yes q summer bridge 
; : ; necessary and not 
County, FL. Available from ProQuest Dissertations and Theses eeeanitehed program 
database (UMI No. 3292594). 
Homel, S. M. (2013). Act 101 summer bridge program: Baseline 
i ; ? Developmental 
An assessment of student success following one-year equivalence is : 
yeaa : i ‘ Yes summer bridge 
participation. Doctoral dissertation. Available from ProQuest necessary and not Roerani 
Dissertations and Theses database (UMI No. 3564815). established pire 
Hudspeth, M. C., & Aldrich, J. W. (2000, June). Summer bridge Baseline 
to engineering. Paper presented at the American Society for Yee equivalence is Summer bridge 
Engineering Education annual conference and exposition, necessary and not program 
Washington, DC. established 
Johnson, A. W. (1998). An evaluation of the long-term impacts Baseline F 
. equivalence is Sponsor-A- 
of the Sponsor-a-Scholar program on student achievement. Yes 
; ; : : necessary and not Scholar 
Princeton, NJ: Mathematica Policy Research. : 
established 
: : : . . Baseline 
Klein, B., & Wright, L. M. (2009). Making pre-algebra meaningful: ; : Developmental 
‘ tune as : equivalence is : 
It starts with faculty inquiry. New Directions for Community Yes summer bridge 
Colleges, 145,67-77 PES sell namo program 
’ ' ‘ established 
Logan, C. R., Salisbury-Glennon, J., & Spence, L. D. (2000). The Baseline 
Learning Edge Academic Program: Toward a community Vale equivalence is Summer bridge 
of learners. Journal of The First-Year Experience & Students in necessary and not program 
Transition, 12(1), 77-104. established 
Malone, M. S. (2014). Persistence and success: Summer bridge Baseline 
program effectiveness. Doctoral dissertation, Johnson & ves equivalence is Summer bridge 
Wales University, Providence, RI. Available from ProQuest necessary and not program 
Dissertations and Theses database (UMI No. 3621984). established 
Maples, S. C. (2002). Academic achievement and retention rate Bacelina 
of students who did and did not participate in a university : : ; 

f ; : : . equivalence is Summer bridge 
summer bridge program. Doctoral dissertation, University of Yes Ae ease aan and bab cerned 
Nevada, Reno, NV. Available from ProQuest Dissertations and ent Prog 
Theses database (UMI No. 3060379). 

Martinez, M. C. G. (2000). The effect of the College Exploration Early acne 
Intervention summer program on performance and retention of seiialeneeie Developmental 
Hispanic disadvantaged first-year students in higher education. Yes ae summer bridge 


Doctoral dissertation. Available from ProQuest Dissertations and 
Theses database (UMI No. 9987240). 


necessary and not 


established 


program 


Nonacademic Interventions for Postsecondary Enrollment and Success with Rural and High-Poverty Populations: 


A Systematic Evidence Review 


36 


Table D2. Studies that met the initial eligibility criteria but were excluded from the report because they did 
not meet What Works Clearinghouse (WWC) standards (continued) 


Reason for not 
meeting WWC 


Reviewed 
Intervention 


Full citation by the 
WWC? standards 


Maye, S. J. (1997). Evaluation of the effectiveness of the Hampton 
University summer bridge program. Doctoral dissertation. 


Baseline 
equivalence is 


Summer bridge 


Available from ProQuest Dissertations and Theses database es necessary and not program 
(UMI No, 9733873). established 
McEvoy, S. (2012). The study of an intervention summer bridge : 
: an ane : Baseline 
program learning community: Remediation, retention, and : ‘ Developmental 
. . : equivalence is . 
graduation. Doctoral dissertation, Claremont Graduate Yes summer bridge 
: : é necessary and not 
University, Claremont, CA. Available from ProQuest established program 
Dissertations and Theses database (UMI No. 3503243). 
McMinn, H. M. (2004). Assessment of the college preparatory Baseline 
program: A prediction model and retention study, 1995-2003. Vas equivalence is Summer bridge 
Doctoral dissertation. Available from ProQuest Dissertations necessary and not program 
and Theses database (UMI No. 3151305). established 
Ohland, M., Zhang, G., Foreman, F., & Haynes, F. (2000, October). The 
Engineering Concepts Institute: The foundation of a comprehensive Baseline 
minority student development program at the FAMU-FSU equivalence is Summer bridge 
; : . : Yes 
College of Engineering. Paper presented at the American Society necessary and not program 
for Engineering Education/Institute of Electrical and Electronics established 
Engineers Frontiers in Education expo, Kansas City, MO. 
Outlaw, J. S. (2008). Academic outcomes of academic success Beeline F . 

f : j equivalence is Summer bridge 
programs. Doctoral dissertation. Available from ProQuest Yes pecessany sadnce rapeach 
Dissertations and Theses database (UMI No. 3319077). nally Prog 

established 
Pike, G. R., Hansen, M. J., & Childress, J. E. (2014). The influence of Baseline 
: ae : : : ‘ Summer 
students’ pre-college characteristics, high school experiences, equivalence is . 

f oe oy Yes bridge 
college expectations, and initial enrollment characteristics on necessary and not Pera 
degree attainment. Journal of College Student Retention, 16(1), 1-23. established prog 

Prather, E. N. (1996). Better than the SAT: A study of the effectiveness of : 
f : Paae Baseline 
an extended bridge program on the academic success of minority eariealeneels Summer 
first-year engineering students. Doctoral dissertation, University of Yes q bridge 
alee gece ; : : necessary and not 
Cincinnati, Cincinnati, OH. Available from ProQuest Dissertations and actablichad program 
Theses database (UMI No. 9622371). 
: ; Baseline 
Robert, E. R., & Thomson, G. (1994). Learning assistance and the equivalence is Summer bridge 
success of underrepresented students at Berkeley. Journal of Yes Hacassary anda captain 
Developmental Education, 17(3), 4-6, 8, 10, 12, 14. sary Prog 
established 
Spurling, S. (2000). The effect of process interventions and Be ecline ‘ 

: ‘ : : equivalence is Summer 
matriculation services on student persistence and success. Yes necessary and not counseling 
Journal of Applied Research in the Community College, 8(1), 31-41. actabliched 

Stephan, J. L. (2010). Closing college enrollment gaps: Whether Baseline 
and how high schools can help. Doctoral dissertation, ree equivalence is Gihiar 
Northwestern University, Evanston, IL. necessary and not 
https://search.proquest.com/docview/753512273 established 
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Table D2. Studies that met the initial eligibility criteria but were excluded from the report because they did 
not meet What Works Clearinghouse (WWC) standards (continued) 


Reason for not 
meeting WWC 
ir-latet-laeksy 


Reviewed 
by the 
WWC? 


Intervention 


Ul | Medit-la(ey a) 


Stewart, J. A. (2006). The effects of a pre-freshman college 


Baseline 


summer program on the academic achievement and ealiivatence-is Summer 
retention of at-risk students. Doctoral dissertation. Available Yes ne sha Ae bridge 
from ProQuest Dissertations and Theses database (UMI No. pesca program 
3208076). 

Torres, J. D., & Cummings, T. (2000, June). Minority education eae Sugarer 
in engineering, mathematics and science. Paper presented Vee eanree be bride 
at American Society for Engineering Education annual airibured Selena - ie 
conference and exposition, Washington, DC. ihe oteeecicn prog 

Vinson, T. L. (2008). The relationship that summer bridge 
and non-summer bridge participation, demographics, pescline 
and high school academic performance have on first- ena alencale Developmental 
year college students: Effects of grade point average and Yes ioe and not Summer bridge 
retention. Doctoral dissertation, Grambling State University, Lea program 
Grambling, LA. Available from ProQuest Dissertations and 
Theses database (UMI No. 3838048). 

Waller, T. O. (2009). A mixed method approach for assessing Baseline earner 
the adjustment of incoming first-year engineering students Ved equivalence is bridee 
ina summer bridge program. Doctoral dissertation, Virginia necessary and not ees . 
Polytechnic Institute and State University, Blacksburg, VA. established pres 

Walpole, M., Simmerman, H., Mack, C., Mills, J. T., Scales, M., Baseline Suinerar 
& Albano, D. (2008). Bridge to success: Insight into summer ex equivalence is binided 
bridge program students’ college transition. journal of The necessary and not bes oa 
First-Year Experience & Students in Transition, 20(1), 11-30. established PIve 

Wheatland, J. A. (2000). The relationship between attendance at 
a summer bridge program and academic performance and pesca 
retention status of first-time freshman science, engineering, eaialalenieale Summer 
and mathematics students at Morgan State University, an Yes ee snclink bridge 
historically Black university. Doctoral dissertation. Available PORE program 
from ProQuest Dissertations and Theses database (UMI No. 

9997415). 

Wolf-Wendel, L. E., Tuttle, K., & Keller-Wolff, C. M. (1999). Baseline Suiniier 
Assessment of a freshman summer transition program vee equivalence is bride 
in an open-admissions institution. journal of The First-Year necessary and not een 


Experience & Students in Transition, 11(2), 7-32. 


established 


Note: WWC is What Works Clearinghouse. FLIGHT is Facilitating Long-Term Improvements in Graduation and Higher Education for Tomorrow. 
Studies cited in the text are also included in the report references section. Studies that were screened but ultimately excluded and not mentioned 
specifically in the report are not in the report references section. 


Source: Authors’ search of ProQuest’s Education Database and the WWC website between June and December 2018 and WWC ratings in January- 
June 2019. The WWC confirmed ratings in July-October 2019. 
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Table D3. Studies that met the initial eligibility criteria but were excluded from the report because they did 
not have statistically significant positive effects on the outcomes of interest 


cae Reviewed by : : 

Full citation the WWC? WWC rating Intervention 

Barnett, E. A., Bork, R. H., Mayer, A. K., Pretlow, J., Wathington, 
H. D., & Weiss, M. J. (2012). Bridging the gap: An impact study Meets WWC Hevelaomental 
of eight developmental summer bridge programs in Texas. Vas standards annie - Bridee 
New York: Columbia University, Teachers College, National without resrarn 8 
Center for Postsecondary Research. reservations Prog 
https://files.eric.ed.gov/fulltext/ED539188.pdf 

Bergin, D. A., Cooks, H. C., & Bergin, C. C. (2007). Effects of Meets WWC 
a college access program for youth underrepresented in Vis standards EXCEL 
higher education: A randomized experiment. Research in without 
Higher Education, 48(6), 727-750. reservations 

Castleman, B. L., Arnold, K., & Wartman, K. L. (2012). Stemming 
the tide of summer melt: An experimental study of the Meets WWC 
effects of post-high school summer intervention on low- Yes standards Summer 
income students’ college enrollment. Journal of Research on without counseling 
Educational Effectiveness, 5(1), 1-17. reservations 
https://eric.ed.gov/?id=EJ]952097 

Castleman, B. L., Owen, L., & Page, L. C. (2015). Report to Meets WWC 
College Bound St. Louis on the implementation and impact of ee standards Summer 
the 2014 summer melt intervention utilizing Bridgit. St. Louis, without counseling 
MO: College Bound. reservations 

Castleman, B. L., Owen, L., & Page, L. C. (2015). Stay late Meets WWC 
or start early? Experimental evidence on the benefits of Yes standards Summer 
college matriculation support from high schools versus without counseling 
colleges. Economics of Education Review, 47, 168-179. reservations 

Castleman, B. L., & Page, L. C. (2015). Summer nudging: Can Meets WWC 
personalized text messages and peer mentor outreach ee andards Seinen 
increase college going among low-income high school Yes withiour Aeuneeln 
graduates? Journal of Economic Behavior and Organization, eabavatiOns g 
115, 144-160. https://eric.ed.gov/?id=E]1124459 

Nathan, A. B. (2013). Does Upward Bound have an effect on ay 
student educational outcomes? A reanalysis of the Horizons recervalions Does 
randomized controlled trial study. Doctoral dissertation, No nokrenartadediiate Upward Bound 
University of Wisconsin-Madison, Madison, WI. ; por q 
https://search.proquest.com/docview/1355756348 information to . 

compute effect sizes 

Rodriguez-Planas, N. (2010). Longer-term impacts of mentoring, Meets WWC 
educational services, and incentives to learn: Evidence from a eeandards Quantum 
randomized trial in the United States. Barcelona Economics Yes without Opportunity 
Working Paper Series. Bellaterra, Spain: Autonomous reeaieanone program 
University of Barcelona. 

Schirm, A., Stuart, E., & McKie, A. (2006). The Quantum —— Quantum 
Opportunity Program demonstration: Final impacts. Yes whet Opportunity 
Washington, DC: Mathematica Policy Research, Inc. program 


reservations 


Note: WWC is What Works Clearinghouse. Studies cited in the text are also included in the report references section. Studies that were screened but 
ultimately excluded and not mentioned specifically in the report are not in the report references section. 


Source: Authors’ search of ProQuest’s Education Database and WWC website between June and December 2018 and WWC ratings in January-June 


2019. The WWC confirmed ratings in July-October 2079. 
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Appendix E: Full description of the interventions discussed in the report 


This appendix describes the studies highlighted in the report. It includes the full citation for each study, the What Works Clearinghouse (WWC) 
rating, a description of the intervention as implemented in the study, the sample, and the reported outcomes. 


Table E1. College counseling 


ie study Intervention description Study design Outcomes 
rating 
Carrell, S. E., & Sacerdote, © Meets WWC College counseling interventions provide Randomized controlled The sample included Postsecondary 
B. (2013). Why do college group design high school seniors with mentoring trial in which eligible 1,150 high school enrollment 
going interventions work? standards and assistance with completing college | students applying for students in New * Positive and 
(NBER Working Paper without applications. In this study, college the college counseling Hampshire. statistically 
No. 19031, issued May reservations undergraduates met with high school program and consenting significant 
2013, revised April 2015). seniors to help them navigate the steps __ to participate in the study ; 
American Economic Journal: involved in applying to college, provided were randomly assigned : ee ; 
Applied Economics, 9(3), feedback on their application essays, to the treatment and improvernent index 
124-151. https://www. and helped them fill out paperwork, comparison groups. The Postsecondary 
nber.org/papers/w19031 including financial aid applications. The | comparison group could persistence 
intervention also included financial use any services the * Indeterminate 
support with application fees anda$100 colleges provided except effects 
incentive for completing the application — for college counseling. 
process. 
WWC is What Works Clearinghouse. 
Source: Carrell & Sacerdote (2013) and WWC (2015). 
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Table E2. Developmental summer bridge program 


WWC study 


i TaicyaY(-Vand(olame(=s-Yerg) ol icey a) 
elilare| 


Study design 


Outcomes 


Barnett, E. A., Bork, R. Meets WWC Offered in summer 2009, students 

H., Mayer, A. K., Pretlow, group design received instruction in one academic 
J., Wathington, H. D., & standards area (math, reading, or writing); 

Weiss, M. J. (2012). Bridging without academic support; a college knowledge 


the gap: An impact study reservations 
of eight developmental 
summer bridge programs in 


Texas. New York: Columbia 


component; and a $400 stipend. 


Randomized controlled 
trial in which eligible 
students applying fora 
developmental summer 
bridge program and 
consenting to participate in 
the study were randomly 
assigned to the treatment 


The sample included 
793 treatment and 
525 comparison 
students. Students 
were primarily 
recent high school 
graduates enrolling 
at two open- 


Credit accumulation 

(postsecondary 

persistence) 

* Indeterminate 
effects 


Academic performance 
* Indeterminate 


iversi ; iti : effects 
University, Teachers and comparison groups. admissions and six 
College, National Center for The comparison group community colleges 
Postsecondary Research could use any services in Texas. 
the colleges provided 
except for summer bridge 
programs. 
WWC is What Works Clearinghouse. 
Source: Barnett et al. (2012) and WWC (2075). 
Table E3. EXCEL 
WWC stud 
Study falng y Intervention description Study design FT e) (=) Outcomes 
Bergin, D. A., Cooks, H. Meets WWC EXCEL is a “scholarship incentive and Randomized controlled The sample included Postsecondary 
C., & Bergin, C. C. (2007). group design = support program” for grade 8 students, _ trial where students 83 students in grade enrollment 
Effects of a college standards sponsored by a midwestern university. were randomly assigned 8:43 inthetreatment + Indeterminate 
access program for youth ~— without It provides parental-involvement to the program (EXCEL) and 40 in the effects 


underrepresented in higher reservations 
education: A randomized 

experiment. Research in 

Higher Education, 48(6), 

727-750. 


programming, weekend enrichment 
seminars for students to build their 
nonacademic skills and cultural 
knowledge, and scholarships to the 
sponsoring university. 


or comparison group, 
stratified by prior 
achievement level, 
gender, and ethnicity. 


comparison group. 
The sample included 
students from low- 
income households, 
but the authors did 
not disaggregate 


results for this group. 


WWC is What Works Clearinghouse. 
Source: Bergin et al. (2007) and WWC (2009). 
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Table E4. Facilitating Long-Term Improvements in Graduation and Higher Education for Tomorrow (FLIGHT) 


WWC study 


f Intervention description Study design Outcomes 
rating 
Philp, J. D. (2015, October). Meets WWC FLIGHT offered a scholarship Randomized controlled 315 students in College access 
FLIGHT: Final evaluation group design and academic services as well as trial in which students grades 7,9, and 11 and enrollment 
report. Columbia, SC: The standards with nonacademic components, including were randomly assigned _ in four Florida school (postsecondary 
Evaluation Group. reservations one-on-one mentoring, case to the treatment and districts enrollment) 
management, and a report of student comparison groups * Positive and 
performance shared with parents, within county and grade. statistically 
teachers, and guidance counselors. significant 
* Size: +39 


improvement index 


WWC is What Works Clearinghouse. 
Note: One student who had not been randomly assigned was included in the analysis, and the analysis did not account for the probability of assignment, compromising the randomized controlled trial. 
Source: Philp (2015) and WWC (20179b). 
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Table E5. Free Application for Federal Student Aid (FAFSA) interventions 


WWC st 
Study C study 

rating 
Bettinger, E., Long, Meets WWC 
B., Oreopoulos, P., & group design 


Sanbonmatsu, L. (2012). 
The role of application 
assistance and information 
in college decisions: Results 
from the H&R Block FAFSA 
experiment. Quarterly 
Journal of Economics, 127(3), 
1205-1242. 


reservations 


Taic-yaY(-Vand(olame(=s-Yerg) ol icey a) 


Two intervention conditions occurred 
immediately after a family filed its taxes 


standards with atalocal H&R Block tax office. The first 


intervention group (FAFSA) received help 
completing and filing the FAFSA and 
identifying potential financial aid amounts 
based on tax information and tuition 
estimates for four local colleges. The 
second intervention group (information- 
only) received only potential financial 

aid and tuition estimates for four local 
colleges, and no assistance in completing 
the FAFSA. The control group received 
neither the FAFSA help nor information 
on financial aid or tuition estimates. All 
three groups received a brochure with 
information on the importance of college 
and general information on college costs 
and financial aid. 


Study design 


Randomized controlled 


trial in which participants 
were randomly assigned 


to one of three groups: 


two intervention groups 
(FAFSA and information- 


only) and one control 


(no-intervention) group. 


FT] o (2) 


The sample for the 
FAFSA intervention 
included 390 
treatment and 398 
comparison group 
participants who 
were 17-year-old 
high school seniors 
or recent high 
school graduates 
dependent on 
their parents, all 

of whose families 
had filed taxes 

at H&R Block tax 
offices in Charlotte, 
North Carolina, and 
throughout Ohio. 


Outcomes 


Postsecondary enrollment 

during the first year 

following experiment 

(postsecondary 

enrollment) 

* Positive and 
statistically 
significant 


* Size: +8 
improvement index 


Retention in college for 
two consecutive years 
following experiment 
(postsecondary 
persistence) 

* Positive and 
statistically 
significant 

* Size: +9 
improvement index 


Page, Lindsay C., Meets WWC 
Castleman, Benjamin, & group design 
Meyer, Katharine. (2018, standards 
May 14). Customized without 


nudging to FAFSA 
completion and income 
verification. Available at 
SSRN (formerly Social 
Science Research Network, 
online): https://ssrn.com/ 
abstract=2854345 or 
http://dx.doi.org/10.2139/ 
ssrn.2854345 


reservations 


Weekly, personalized text messages 
related to FAFSA and financial aid. 
Included general and customized 
messages based on students’ FAFSA- 
filling status. 


School-level randomized 


controlled trial that 
randomly assigned 39 
schools to the FAFSA 


texting intervention and 


27 to business as usual. 


The sample included 
39 treatment and 27 
comparison schools 
across eight school 
districts in the area 
of Houston and 
Austin, Texas. 


Full-time 

postsecondary 

enrollment 

(postsecondary 

enrollment) 

* Positive and 
statistically 
significant 


* Size: +7 
improvement index 


WWC is What Works Clearinghouse. 


Source: Bettinger et al. (2012); Page et al. (2018); WWC (2013, 201 9a). 
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Table E6. Quantum Opportunity Program (QOP) 


Study Ne study 
rating 
Schirm, A., Stuart, E., Meets WWC 
& McKie, A. (2006). The group design 
Quantum Opportunity standards 
Program demonstration: without 


Final impacts. Washington, reservations 
DC: Mathematica Policy 


Research, Inc. 


TaicyaY(-Vand(olame(=s-Yerg) lice) a) 


The Quantum Opportunity Program 
(QOP) provided mentoring and 
case management, life skills and 
employment-readiness courses, a 
community service component, and 


financial incentives to grade 9 students. 


Students could be enrolled in the 
program for up to five years. 


cS) 0 Te MV Ao (-X-3 (ea) 


Randomized controlled 
trial in which students 
were randomly assigned 
to either QOP ora 
business-as-usual control 
group. The students 
included in the study 
were randomly selected 
from a pool of all eligible 
grade 9 students in each 
of the study schools. 


Sample 


The study 

included 1,069 

(580 in treatment) 
students in grade 

9 in Cleveland, 

Ohio; Fort Worth, 
Texas; Houston, 
Texas; Memphis, 
Tennessee; 
Philadelphia, 
Pennsylvania; 
Washington, DC; and 
Yakima, Washington. 


Outcomes 


Postsecondary 

enrollment 

* Indeterminate 
effects 


Earned a degree/ 

certificate 

(postsecondary 

completion) 

* Indeterminate 
effects 


Rodriguez-Planas, N. Meets WWC 
(2010). Longer-term impacts group design 
of mentoring, educational standards 
services, and incentives without 


to learn: Evidence from reservations 
a randomized trial in the 

United States. Barcelona 

Economics Working Paper 

Series. Bellaterra, Spain: 

Autonomous University of 


Barcelona. 


This follow-up study of the QOP 
examined the same intervention as the 
Schirm, Stuart, and McKie (2006) study, 
following up with students at the end 
of the program and two and five years 
after the end of the program. 


Same as above 


Same as above, 
with three follow- 
up periods: at age 
19 (the end of the 
program), 21 (two 
years after the 
program), and 24 
(five years after the 
program 


Postsecondary 

enrollment 

* Indeterminate 
effects 


Earned a degree/ 

certificate 

(postsecondary 

completion) 

* Indeterminate 
effects 


WWC is What Works Clearinghouse. 


Source: Rodriguez-Planas (2010); Schirm et al. (2006); WWC (2016b, 2017). 
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Table E7. Summer bridge program 


WWC stud 
y TaiyaY(-land(olame(=s-Yerg) ol icey a) 


S10 Te MY Ao (-X-3(e a) 


Outcomes 


rating 


Murphy, T. E., Gaughan, M., Meets WWC Five-week program that included Quasi-experimental The sample included Graduation rates 
Hume, R., & Moore, S. G. group design anonacademic peer-mentoring design with atreatment 770 treatment and (postsecondary 
(2010). College graduation standards with component. group of students who 1,452 comparison completion) 
rates for minority reservations enrolled inthe summer group freshmen ata _ ‘= Positive and 
students in a selective bridge program anda technical university statistically 
technical university: Will comparison group of in the southeastern significant 
participation in a summer students who chose not __ United States. * Size: +4 
bridge program contribute to participate. improvement index 
to success? Educational 
Evaluation and Policy 
Analysis, 32(1), 70-83. 
WWC is What Works Clearinghouse. 
Source: Murphy, et al. (2010); WWC (2007, 2009a). 
Table E8. Summer counseling 
Study ba eiudy Intervention description Study design FT] oy (2) Outcomes 
Castleman, B. L., Arnold, Meets WWC Counseling services supported Randomized controlled The sample included Percentage of students 
K., & Wartman, K. L. group design — students in obtaining financial aid and __ trial in which students 80 treatment and 82. who enrolled in college 
(2012). Stemming the standards addressing socioemotional concerns. were randomly assigned comparison group in the fall after high 
tide of summer melt: An without to receive summer graduating seniors in school graduation 


counseling or no 
counseling. 


experimental study of the reservations 
effects of post-high school 

summer intervention on 

low-income students’ 

college enrollment. Journal 

of Research on Educational 


Effectiveness, 5(1), 1-17. 


seven high schools (postsecondary 

in Rhode Island in enrollment) 

2008. * Indeterminate 
effects 
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Table E8. Summer counseling (continued) 


Study 


Castleman, B. L., Owen, L., 
& Page, L. C. (2015). Report 
to College Bound St. Louis 
on the implementation and 
impact of the 2014 summer 
melt intervention utilizing 
Bridgit. St. Louis, MO: 
College Bound. 


WWC study 
e-ilare| 

Meets WWC 
group design 
standards 
without 
reservations 


Intervention description 


The Bridgit platform—a schoolwide 
intervention tracking students’ progress 
toward college enrollment—was 
implemented in summer 2014. Based 
on student-created profiles indicating 
college attendance plans, a set of 
prioritized pre-enrollment steps was 
created by the system. A counselor was 
assigned to each student to monitor 
student progress and follow up via text 
message or phone as needed. 


Study design 


Randomized controlled 
trial in which researchers 
randomly assigned 
matched pairs of 
schools within districts 
to the intervention or 
comparison group. 
Researchers matched 
schools based on the 
prior year’s college 
enrollment rate. 


FT l oy (=) 


The sample included 
10 schools from 
Shelby County in 
Tennessee and 

two schools from 
Ferguson-Florissant 
in Missouri. Half 
the schools from 
each district were 
assigned to the 
intervention group 
and half to the 
comparison group. 
At least 60 percent 
of students at each 
school were eligible 
for free or reduced- 
priced lunch. 


Outcomes 


College access 

and enrollment 

(postsecondary 

enrollment) 

* Indeterminate 
effects 


Castleman, B. L., Owen, 
L., & Page, L. C. (2015). 
Stay late or start early? 
Experimental evidence 
on the benefits of college 
matriculation support 
from high schools versus 
colleges. Economics of 
Education Review, 47, 
168-179. 


Meets WWC 
group design 
standards 
without 
reservations 


Twenty-one counselors—eight based 

at the University of New Mexico 

(UNM) and 14 based at high schools 

in Albuquerque Public Schools (APS)— 
reached out to students admitted to 
UNM to support completion of summer 
tasks needed for college enrollment 

in the fall. Counselors reached out via 
phone, email, and text message for five 
to six weeks over the summer. 


Randomized controlled 
trial in which students 
were randomized 
within each high 
school into three 
groups: (1) intervention 
group assigned to a 
UNM counselor; (2) 
intervention group 
assigned to an APS 
counselor; and (3) 
comparison group. 


Sample included two 
treatment groups, 
those assigned to 

a UNM counselor 

(N = 528) and those 
assigned to an APS 
counselor (N = 539), 
and one comparison 
group (N = 528). 


College access 

and enrollment 

(postsecondary 

enrollment) 

* Indeterminate 
effects 
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Table E8. Summer counseling (continued) 


Intervention description 


Study design 


Outcomes 


WWC study 

elilare) 
Castleman, B. L., & Page, L. Meets WWC 
C. (2015). Summer nudging: group design 
Can personalized text standards 
messages and peer mentor without 


outreach increase college reservations 
going among low-income 

high school graduates? 

Journal of Economic 

Behavior and Organization, 


115, 144-160 


Study examined two summer 
counseling interventions: (a) automated 
text messaging and (b) peer mentoring. 
As part of the automated text- 
messaging intervention, students and 
their parents received 10 automated 
text messages about college enrollment 
tasks (for example, register for 
placement test and housing), along with 
a prompt to request help if needed. 
Students were linked to counselors 

or financial aid advisers for help. 
Students in the peer-mentoring group 
were contacted by peer mentors who 
assessed students’ readiness for college 
in the fall and discussed topics such as 
plans for enrolling in the fall and FAFSA 
completion. 


Randomized controlled 
trial where students 
were randomly assigned 
within their district to 
one of the two summer 
counseling interventions 
or a comparison 

group. In Philadelphia, 
students were randomly 
assigned to the peer- 
mentor intervention 

or the comparison 
group. In Dallas, 
students were randomly 
assigned within each 
counselor's cluster to 
the texting intervention 
or comparison group. 

In Boston, Lawrence, 
and Springfield, 
Massachusetts students 
were randomly assigned 
to the automated 
text-messaging, 

peer mentoring, or 
comparison group. 


The sample 
included college- 
intending high 
school graduates 
across five sites. 
The peer-mentoring 
intervention had 
934 students in 

the intervention 
group and 1,272 
students in the 
comparison groups 
from Philadelphia, 
Boston, Lawrence, 
Massachusetts; 
and Springfield, 
Massachusetts. The 
texting intervention 
had 2,524 students 
in the intervention 
group and 2,535 
students in the 
comparison group 
from Dallas, 
Boston, Lawrence, 
Massachusetts; 
and Springfield, 
Massachusetts. 


College access 

and enrollment 

(postsecondary 

enrollment) 

* Indeterminate 
effects 
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Table E8. Summer counseling (continued) 


Study bade emuay, Intervention description Study design Sample Outcomes 

Castleman, B. L., Page, L. Meets WWC Counseling services supported Randomized controlled The sample included Percentage of students 
C., & Schooley, K. (2014). group design — students in obtaining financial aid and _ trial in which eligible 886 treatment and who enrolled in college 
The forgotten summer: standards addressing socioemotional concerns. students were randomly 1,487 comparison in the fall after high 
Does the offer of college without assigned to receive group students from — school graduation 
counseling after high reservations summer counseling or the class of 2011 (postsecondary 

school mitigate summer no counseling. in Fulton County, enrollment) 

melt among college- Georgia, and Boston, « Positive and 
intending, low-income Massachusetts. statistically 

high school graduates? significant 

Journal of Policy Analysis » Size: +6 


and Management, 33(2), improvement index 
320-344. 

Percentage of students 

who remained enrolled 

in college after the 
spring of freshman 
year and after fall 

of sophomore year 

(postsecondary 

persistence) 

* Positive and 
statistically 
significant 

* Size: +6 
improvement index 


FAFSA is Free Application for Federal Student Aid. WWC is What Works Clearinghouse. 
Source: Castleman et al. (2012); Castleman et al. (2015a, 2015b); Castleman & Page (2015); Castleman et al. (2014); and WWC (2018). 
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Table E9. Upward Bound 


ee auey. Intervention description Study design Outcomes 
rating 
Myers, D., Olsen, R., Seftor, Meets WWC Upward Bound is a federal program Random assignment About 1,500 students College credits earned 
N., Young, J., & Tuttle, group design that provides high school students from — study. Applicants were assigned to (postsecondary 
C. (2004). The impacts of standards low-income households and who are to set of nationally the intervention and _ persistence) 
regular Upward Bound: without first-generation college students with representative Upward about 1,300 students . |ndeterminate 
Results from the third reservations academic and nonacademic supports, Bound projects were were assigned to the effects 
follow-up data collection. such as cultural activities, college tours, randomly assigned to comparison group 
Princeton, NJ: Mathematica and help with FAFSA completion, all with the intervention or a across 67 nationally 
Policy Research the aim of increasing postsecondary comparison group. representative 
enrollment and completion. Upward Bound 
projects. 
Seftor, N. S., Mamun, A., Meets WWC This follow-up study of Upward Bound Same as above Same as above Postsecondary 
& Schirm, A. (2009). The group design examined the same intervention as the enrollment 
impacts of regular Upward standards Myers et al. (2004) study, following up * Indeterminate 
Bound on postsecondary without with students seven to nine years after effects 
outcomes 7-9 years after reservations high school graduation. ; 
scheduled high school College credits earned 
(postsecondary 


graduation. Princeton, 
NJ: Mathematica Policy 
Research. 


persistence) 
* Indeterminate 
effects 


Earning a bachelor’s 
or associate degree 


(postsecondary 
completion) 
* Indeterminate 
effects 
FAFSA is Free Application for Federal Student Aid. WWC is What Works Clearinghouse. 
Source: Myers et al. (2004); Seftor et al. (2009); WWC (2009b, 2009c). 
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